{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A fastai/Pytorch implementation of MixMatch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this post, I will be discussing and implementing \"MixMatch: A Holistic Approach to Semi-Supervised Learning\", by Berthelot, Carlini, Goodfellow, Oliver, Papernot and Raffel [1]. Released in May 2019, MixMatch is a semi-supervised learning algorithm which has significantly outperformed previous approaches. This blog comes from discussions had within Dr. Ehsan Kamalinejad's machine learning research group at Cal State East Bay.\n",
    "\n",
    "How much of an improvement is MixMatch? When trained on CIFAR10 with 250 labeled images, MixMatch outperforms the next best technique (Virtual Adversarial Training) by almost 25% on the error rate (11.08% vs 36.03%, for comparison the fully supervised case on all 50k images has an error rate of 4.13%). These are far from incremental results, and the technique shows the potential to dramatically improve the state of semi-supervised learning. \n",
    "\n",
    "Semi-supervised learning is largely a battle against overfitting; when the labeled set is small it doesn't take a very large neural network to memorize the entire training set. The general idea behind nearly all semi-supervised approaches is to leverage unlabeled data as a regularizer on the training of labeled data. For a great overview of various semi-supervised learning methods, see [this blog by Sebastian Ruder](http://ruder.io/semi-supervised/). Different techniques employ different forms of regularization, and the MixMatch paper divides these into three groups: entropy minimization, consistency regularization, and generic regularization. As all three forms of regularization have proved effective, the MixMatch algorithm contains features from each.\n",
    "\n",
    "MixMatch is a combination and improvement upon several of these techniques that have come out in recent years, including: Mean Teacher [2], Virtual Adversarial Training [3], and Mixup [4]. At a high level, the idea of MixMatch is to label the unlabeled data using predictions from the model and then apply heavy regularization in several forms. The first is performing data augmentation several times and taking the average for the label prediction. These predictions are then 'sharpened' to reduce their entropy. Finally, Mixup is performed on the labeled and unlabeled sets.\n",
    "\n",
    "I am aiming this post at those familiar with Pytorch, but not necessarily fastai. For a Jupyter notebook version of this post containing the full code needed to reproduce all the results see [this repository](https://github.com/noachr/MixMatch-fastai).\n",
    "\n",
    "### fastai\n",
    "Before diving into the paper, I'll briefly talk about fastai. Fastai is a library, built on Pytorch, which makes writing machine learning applications much easier and simpler. They also offer a [terrific online course](https://course.fast.ai/) covering both fastai and deep learning in general. Compared to pure Pytorch, fastai dramatically reduces the amount of boilerplate code required to produce state of the art neural networks. Here we'll be using the data pipeline and training loop features of fastai. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing fastai will also import numpy, pytorch, etc. \n",
    "from fastai.vision import *\n",
    "from numbers import Integral\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Components\n",
    "Let's first describe the individual pieces needed to assemble MixMatch, and then at the end put them together to form the complete algorithm. Following the paper, we'll be using CIFAR10 and taking 500 randomly selected images as the labeled training set. The standard 10000 image test set is used for all accuracy measurements.  \n",
    "\n",
    "#### Data Augmentation \n",
    "Data augmention is a widely used consistency regularization technique, with it's biggest success (so far) found in the computer vision realm. The idea is to alter the input data while preserving its semantic label. For images, common augmentations include rotation, cropping, zooming, brightning, etc. -- all transformations which do not change the underlying content of the image. MixMatch takes this a step further by performing augmentation multiple times to produce multiple new images. The predictions of the model on these images are then averaged to produce a target for the unlabeled data. This makes the predictions more robust than using a single image. The authors found that just two augments were sufficient to see this benefit.\n",
    "\n",
    "Fastai has an efficient transformation system which we'll utilize on the data. However, as its designed to produce only one augmentation per image and we will need several, we will start by modifying the default LabelList to emit multiple augments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Modified from \n",
    "K=2\n",
    "class MultiTransformLabelList(LabelList):\n",
    "    def __getitem__(self,idxs:Union[int,np.ndarray])->'LabelList':\n",
    "        \"return a single (x, y) if `idxs` is an integer or a new `LabelList` object if `idxs` is a range.\"\n",
    "        idxs = try_int(idxs)\n",
    "        if isinstance(idxs, Integral):\n",
    "            if self.item is None: x,y = self.x[idxs],self.y[idxs]\n",
    "            else:                 x,y = self.item   ,0\n",
    "            if self.tfms or self.tfmargs:\n",
    "                #I've changed this line to return a list of augmented images\n",
    "                x = [x.apply_tfms(self.tfms, **self.tfmargs) for _ in range(K)]\n",
    "            if hasattr(self, 'tfms_y') and self.tfm_y and self.item is None:\n",
    "                y = y.apply_tfms(self.tfms_y, **{**self.tfmargs_y, 'do_resolve':False})\n",
    "            if y is None: y=0\n",
    "            return x,y\n",
    "        else: return self.new(self.x[idxs], self.y[idxs])\n",
    "        \n",
    "#I'll also need to change the default collate function to accomodate multiple augments\n",
    "def MixmatchCollate(batch):\n",
    "    batch = to_data(batch)\n",
    "    if isinstance(batch[0][0],list):\n",
    "        batch = [[torch.stack(s[0]),s[1]] for s in batch]\n",
    "    return torch.utils.data.dataloader.default_collate(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai's [data block api](https://docs.fast.ai/data_block.html) allows for flexibly loading, labeling, and collating nearly any form of data. However, it doesn't have a method to grab a subset of one folder and the entirety of another folder, which is required here. Thus, we'll subclass the ImageList class and add a custom method. We'll use fastai's `get_transforms` method with no arguments to use the default image transforms; these are flipping around the center y axis, rotation up to 10 degrees, zooming, lighting change, and warping. Fastai's transform system automatically randomizes the exact parameters of each transform when applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Grab file path to cifar dataset. Will download data if not present\n",
    "path = untar_data(URLs.CIFAR)\n",
    "\n",
    "#Custom ImageList with filter function\n",
    "class MixMatchImageList(ImageList):\n",
    "    def filter_train(self,num_items,seed=2343):\n",
    "        train_idxs = np.array([i for i,o in enumerate(self.items) if Path(o).parts[-3] != \"test\"])\n",
    "        valid_idxs = np.array([i for i,o in enumerate(self.items) if Path(o).parts[-3] == \"test\"])\n",
    "        np.random.seed(seed)\n",
    "        keep_idxs = np.random.choice(train_idxs,num_items,replace=False)\n",
    "        self.items = np.array([o for i,o in enumerate(self.items) if i in np.concatenate([keep_idxs,valid_idxs])])\n",
    "        return self\n",
    "    \n",
    "#Create two databunch objects for the labeled and unlabled images. A fastai databunch is a container for train, validation, and\n",
    "#test dataloaders which automatically processes transforms and puts the data on the gpu.\n",
    "data_labeled = (MixMatchImageList.from_folder(path)\n",
    "                .filter_train(500) #Use 500 labeled images for traning\n",
    "                .split_by_folder(valid=\"test\") #test on all 10000 images in test set\n",
    "                .label_from_folder()\n",
    "                .transform(get_transforms(),size=32)\n",
    "                #On windows, must set num_workers=0. Otherwise, remove the argument for a potential performance improvement\n",
    "                .databunch(bs=64,num_workers=0)\n",
    "                .normalize(cifar_stats))\n",
    "\n",
    "train_set = set(data_labeled.train_ds.x.items)\n",
    "src = (ImageList.from_folder(path)\n",
    "        .filter_by_func(lambda x: x not in train_set)\n",
    "        .split_by_folder(valid=\"test\"))\n",
    "src.train._label_list = MultiTransformLabelList\n",
    "data_unlabeled = (src.label_from_folder()\n",
    "         .transform(get_transforms(),size=32)\n",
    "         .databunch(bs=128,collate_fn=MixmatchCollate,num_workers=0)\n",
    "         .normalize(cifar_stats))\n",
    "\n",
    "#Databunch with all 50k images labeled, for baseline\n",
    "data_full = (ImageList.from_folder(path)\n",
    "        .split_by_folder(valid=\"test\")\n",
    "        .label_from_folder()\n",
    "        .transform(get_transforms(),size=32)\n",
    "        .databunch(bs=128,num_workers=0)\n",
    "        .normalize(cifar_stats))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mixup \n",
    "\n",
    "$$\n",
    "\\lambda \\sim Beta(\\alpha,\\alpha)\\\\\n",
    "\\lambda' = max(\\lambda,1-\\lambda)\\\\\n",
    "Mixup(a,b) = \\lambda'*a + (1-\\lambda')*b$$\n",
    "\n",
    "Mixup was first introduced by Zhang, Cisse, Dauphin, and Lopez-Paz [4] in 2018 and and falls into the category of general or traditional regularization. Instead of passing single images to the model, Mixup performs a linear interpolation between two seperate training images and passes that to the model. The one hot encoded labels of the images are also interpolated, using the same $\\lambda$ coefficient as the images. That coefficient is randomly drawn from the beta distribution, parameterized by the alpha. Typically, $\\alpha$ needs to be tuned to the dataset. At small values of $\\alpha$, the beta distrubtion has most of it's weight in the tails, close to 0 or 1. As $\\alpha$ increases, the distributions becomes uniform and then increasingly spiked around .5. Thus, $\\alpha$ can be seen as controlling the intensity of the mixup; small values result in only a small amount of mixup, while larger values bias towards maximum mixup (50/50). At the extremes, $\\alpha=0$ results in no mixup at all; and as $\\alpha\\rightarrow\\infty$, $\\beta$ approaches a Dirac delta distribution centered at 0.5. The authors recommend starting with a value of .75, which as seen below still has most of the weight in the tails. The paper makes one modification to the original method, which is to set $\\lambda$ to $max(\\lambda,1-\\lambda)$; this biases the mixup towards the original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAACAEAAAFTCAYAAABC54eEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xl8XPV1///3mRnNSCPJiyx5wRiMWQwEghMMOISQBfoLSQmk34aWtmmgJaE0TdI2/f6+36TfNnv3pmua5EuWhiT9kT0NIUkb0oTQEDZDwCwGY2xiG4wlWbItaaQZzcz5/TH3CiHLtpbZ7tXr+XjooVk+c+8Zg+/1zDn3HHN3AQAAAAAAAAAAAACA6Es0OgAAAAAAAAAAAAAAAFAdFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAAAAAAAAABATFAEAAAAAAOrOzHrM7ONm9rSZ5czsCTN76zTr/tnMvt2IGCfF8IdmtsXMZvQZ2syuNTOf9DMSvM9vmtmvTN2OmX3AzHyWMb3RzN49y9e8YD/hfTNLzWY7c4lpLu8RAAAAAADMDUUAAAAAAIC6MrOMpO9KukzS+yVdLmmzpP9rZmsmrTtZ0u9I+mAj4pzkk5KWS7pmlq+7StLLJL1e0p9Kyku6WdL3zaxt0rpPB+tm442SZlUEMMf9zMbRYqr1vgEAAAAAQKAq1f4AAAAAAMzCmySdK+lsd39UksysIOnXJbVPWvcHkh5y9831D/F57j5qZp+X9D8l/essXvqgu2+fdP8LZvZVSV+V9NeS3hlsf4+kPdWKdyozy7h7vtb7OZpG7hsAAAAAgIWGTgAAAAAAgHr7JUmbwwKAwC9L6pf0pDTRLeDNkv6/qS82sxYz+xMz22Fmo2Z2l5mtN7ONwWiB42YayCy29SVJZ5rZhXN905Lk7l+X9C1JbzOzbBDD1Db9pwWjA3rNbMzMdpnZV8O2/Wb2OVW6EqyeNHLg6cnbMrOzzOw/zWxY0lem288kZ5jZj4L3u9fMPjR5ZIGZfS7c/pQ/u9vN7PZjxXSkfZvZZcGf96iZHTSzfzez9VPWhO/nVDP7jpkNm9nPzex9Mx3PAAAAAADAQkMnAAAAAABAvW2S9HUzS0s6QdJvSfp9Sb/j7qVJa5ZI+u/JLzSzFknfk/RiSe+VtE/SxyV9RFKrpH9x92dnEsQst/WgpEOqjDD46ezf8gt8V5XW+Rsl3THN87dKOiDpd1UpjFitykiBMOn9YUk9ks6TdEXwWH7KNr4l6TOS/kpS+Rjx/Lukz0r6C0mvVWV0QVnSB2b4fmYa0wQzu0zSdyT9UNKvSuqQ9CFJPzGzDe7+zJSXfFOVLgx/L+kNqoyI2K3ZdWYAAAAAAGBBoAgAAAAAAFA3ZrZClaT2ZklfkPQrwVNfUSVpHdokySVtmbKJt0t6jaSL3P2nwTbPU6WIwCRdO4twZrwtdy+b2ZYgrvnaFfxeNfUJM+uWdKqkK939lklPTXREcPenzKxPUsHd7z7CPv7J3f9xhvF8yt3/Mrj9fTNbJOmPzOwf3P3ATDYww5gm+4ikHZJe5+5FSTKzuyRtk/RHkt49Zf1H3T1M+P/AzF4j6ddEEQAAAAAAAIehdR4AAAAAoJ42Br/vU+Xq8Ssl3SjpKkl/N2ndcZIOuXthyutvkPT9MGkfOCBpsaS/c/f9s4hlttvqC+KaLwt+T9eaf78qyfG/NLO3mdmpc9zHN2ex9itT7n9JlSvzz5rjvo/KzNolvVTSl8MCAEly952S7pT0ymle9p0p9x9RpYsEAAAAAACYgiIAAAAAAEA9bVSlrf4T7v6Iu9/i7r8j6WZV2sKHWjWlnbyZrZR0uirt9CdLSxrQC4sIjmqO2xqV1DbTfRzFmuD33qlPuLtL+gVVOiX8haRtZrbDzH53lvs4bNtHse8I91fPcp8ztVSVQojpYnxOUtc0jw9MuZ9X5f8RAAAAAAAwBUUAAAAAAIB62ijpgSDZPZmrkgAO7VclWTzZKcHvneEDZpaU9BZJT7n70CzimMu2uiT1z2IfR/KLksYk3T/dk+6+w93fIqlH0ksk/VDSx83sdbPYx3RdBo5kxRHuPxP8HlOlOGKqZbPYx2SDqsS3cprnVqry3x4AAAAAAMwRRQAAAAAAgHo6Vy9M9svMVki6XNLXJj38uKQWMzt+0mPl4PfkK8V/V9KZkpKzjGMu2zpJ0hOz3M8LmNn/kHSFpE+6e+5oa73iQUnvDh6a3J4/r+p0JZCkX5ly/2pJw6q03Jekn0taYWbd4QIzO1nS+imvm1FM7j6iSgHEVUHhRbjNEyVdKOnHs30DAAAAAADgealGBwAAAAAAWBjMbLWkVZJ+xcx2Svq+Ki3n3ytpj6R/nLT8juD3+cFzkvSQpIOS3mdmByUdL+mvJX1d0huCK+V/5O5jwf7WqnKl/wfd/QNTwpnttpZIOk3S387iLW8IEudpVebXXy7pKkm3Be/5MGb24uDP4cuStqtSkHCtpKIqHQFCj0nqCsYEbJY05u4PzyK2yd5mZglJ90l6raS3SvqAux8Inv+qpA9L+jcz+ztJ3UH8U7sizCamP5X0HUm3mtnHJXVI+qAq/00+Osf3AQAAAAAARBEAAAAAAKB+Nga/P69KovmPVLnK/FZJf+7uw+FCd3/azO6V9AZJ3wgeGzGzqyT9vaSvSOqT9DZJ/6lKkv1WSYsm7a89+P2CzgNz3NYvSipI+uYs3u9Xg99jknolPaDKVfZfm2YcQug5SbtUufr/+OC1D0u63N0njw/4tKRNkv5c0hJV/hzXziK2ya6U9M+qJOYPSvqIKkl/SZK7bzezNwWP/7ukbUF8fzxlOzOOyd3/w8x+UdL7VfnzL0i6XdL/cvdn5/g+AAAAAACAJDvy9w4AAAAAAFSPmX1Y0jslLT1KEnzy+mtVuSp+1bFa5x/h9ddL+jNJJ87l9VO29T1J/e7+m/PZDgAAAAAAQK0lGh0AAAAAAGDB2Cjp/pkUAAS+IOkZSW+f4/5eKenvq1AAsEHSq1VpVw8AAAAAANDUGAcAAAAAAKiXcyX960wXu3vJzH5b0kvnsjN3/425vG4aKyX9lrtvr9L2AAAAAAAAaoZxAAAAAAAAAAAAAAAAxATjAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAIA5MLMuM/ummY2Y2c/N7NePsjZjZp8J1g2Z2c/M7HX1jBcAEE2zOd8E64en/JTM7J+D5243s7FJzz1Rn3cBAIgjM3uHmW02s7yZfa7R8QAA4ulY55vZfmYCAGCq+ZxrOA+hmaUaHQAQUf8iqSBphaQNkr5jZg+5+6PTrE1J2i3plZJ2SXq9pK+Y2dnu/nSd4gUARNNszjdy947wtpm1S9on6auTlrzD3T9dw3gBAAvHs5I+Ium1ktoaHAsAIL6Odb6Z1WcmAACmMZ9zDechNC1z90bHANSMmaUkvVfSdZI6Jb1T0vGSWtz9z+a4zXZJg5LOcvdtwWNfkPSMu79nhtvYIumD7v71ucQAAGguzXi+MbNrJL1f0snu7mZ2u6QvUgQAAAtLLc5RU7b/EUnHu/u1890WACC6GnG+qcZ3dACA6Gi2c42kDx/pOc5DaAZ0AkDcfUTSRknnSLpY0l9LckkXTF5kZrdKuugI2/iJu18+6f5pkkrhQT3wkCpX+h+Tma0ItkElGADER9OdbyRdI+nz/sKKz78ws7+U9ISk/+Put89wWwCA6KrFOQoAgKkacb6Z72cmAEC0NNu5hvMQmhpFAIgtM1sk6Q8knenuB83sHkmnq5L0GJq8dpYH/Q5JB6c8dlCVyrNjxdQi6d8k3eTuj89inwCAJtWk55sTVPnAcd2kh/+3pMdUaVF2taRvm9kGd39qFjEBACKkhucoAAAmNPB8M+fPTACAaGnScw3nITS1RKMDAGroNZK2ufuO4H5alQPwP89zu8OSFk15bJGkoWnWTjCzhKQvqJJ8ecc8YwAANI+mOt8E3qJKZfPO8AF3v8fdh9w97+43SbpT0uvnGSMAoLnV6hwFAMBkjTrfzOczEwAgWprxXMN5CE2NIgDE2XGSnp10/3pVZrEcdgA2s++Z2fARfr43Zfk2SSkzO3XSY+foKO39zcwkfUbSCkm/7O7jc31TAICm0zTnm0neIummY6xxSTaDbQEAoqtW5ygAACZr1PlmPp+ZAADR0oznGs5DaGqMA0Cc7ZG0wcxWSTpB0m9K6jCztLsXJi9099fNdKPuPmJm35D0ITN7q6QNkq6UdOFRXvYJSWdIutTdR2f5PgAAza2ZzjcyswslrZb01UmPLVFlPtqPJRUl/aoqs9P+YKbxAAAiqSbnKEkys5Qq3ykkJSXNrFVS0d2L1QkdABAhDTnfzPUzEwAgkpruXMN5CM2OTgCIs/+Q9H1JWyXdLOl/SHpQ0g+rsO23S2qT1Bts+3fdfaK6K6g0++Pg9omSfkeVE8BzkyrOfqMKcQAAGq8pzjeTXCPpG1MqoVskfURSn6R+Se+U9EZ3f6IKMQIAmlctz1F/ImlU0nskvTm4/SdV2C4AIHoaeb456mcmAEBsNOu5hvMQmpa5e6NjAAAAAAAAAAAAAAAAVUAnAAAAAAAAAAAAAAAAYoIiAAAAAAAAAAAAAAAAYoIiAAAAAAAAAAAAAAAAYoIiAAAAAAAAAAAAAAAAYoIiAAAAAAAAAAAAAAAAYiLVqB13d3f72rVrG7V7AFgQ7r///n5372l0HM2E8w8A1BbnnsNx7gGA2uP8czjOPwBQW5x7psf5BwBqa6bnn4YVAaxdu1abN29u1O4BYEEws583OoZmw/kHAGqLc8/hOPcAQO1x/jkc5x8AqC3OPdPj/AMAtTXT8w/jAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAAAAAAAAiAmKAAAAAAAAAIAGMbNWM7vXzB4ys0fN7IPTrLnWzPrM7MHg562NiBUAAABANKQaHcBcfO/hvWpLJ/Wq9csbHQoA1MxX7tutU1Z06KUnLG10KJC0fziv7z+2T684tVvHL802OhwAqIld+3O686l+ve6slVqSTTc6HAAAFoq8pNe4+7CZtUj6iZl9z93vnrLuy+7+jgbEB9RMvljSbY/t09BYUZJkkjatW6a13e2NDQwAEGv7h/P6wdZ9Knvl/uK2Fr3urJUys8YGBlRRJIsAPvaj7Vq5qJUiAACx9v5bHtWbN51AEUCT2HtwTO/9xsP65JvPpQgAQGz9bPeg3vuNh3Xe2qUUAQAAUCfu7pKGg7stwY83LiKg9spl160P79Xf/Ofj2j0w+oLnUgnTb1xwgt55yanq7sg0KEIAQJzdeMcO/d87drzgsVvfeZHOWr24QREB1RfJIoB0KqFCqdzoMACgpgqlstIpprY0i/ZM5ZQ5Ol5scCQAUDujhZIkKZuO5McEAAAiy8ySku6XdIqkf3H3e6ZZ9stmdrGkbZL+0N131zNGoFp27c/pHTc/oC17Dur0lZ3612vP0xmrFkmScoWiPvOTnfriPbv09Qee0Xtff7p+44ITGxwxACBuHtt7SKev7NTnfut87RnM6U2fvEtb9x6iCACxEslv99LJhPJFigAAxFep7CqVXelkstGhINCervy3GMmXGhwJANTOSFAE0E4RAAAAdeXuJUkbzGyJpG+a2Vnu/sikJd+WdLO7583sBkk3SXrN1O2Y2fWSrpekE044oQ6RA7PTe2hMb/7MPTo4Oq6PXnWO3viS1UomXth6+c9+6Wz99kUn6QO3PKr/881HlEqYfvU8/n8GAFTPtn1Devkp3Vq5uFU9nRmlUwlt2zfU6LCAqorkJabpVEIFigAAxFh4jKMTQPPIBp0AcgU6AQCIr1y+coxrS1OEBgBAI7j7AUm3S7psyuP73T0f3P2UpHOP8Pob3X2ju2/s6empaazAbB3Mjestn71X/cN53fTb5+uXzz3+sAKA0Mk9HfrMNefp4tN69N5vPKz/eGRvnaMFAMTVgVxB+w7ltX5FpyQpmTCdurxDT+wbPsYrgWiJZHYpk0rSCQBArOWLlSsxMxQBNI22lkpCLFegEwCA+MqNl9SSNIrQAACoIzPrCToAyMzaJF0q6fEpa1ZNunuFpK31ixCYv9FCSb990316qm9YN/7mRm1Ys+SYr0mnEvrkm1+qDWuW6F03P6ifbu+vQ6QAgLjbFiT7T1vZOfHY+hWd2vYcnQAQL5H8di+TSqhQJAkDIL7oBNB8kglTa0uCIgAAsZbLF5VlFAAAAPW2StKPzGyLpPsk3ebut5rZh8zsimDNu8zsUTN7SNK7JF3boFiBOfngtx/VA7sG9Y9Xv0QXndo949dl0yl99trzdFJ3u37ni/erd2ishlECABaCJ4K2/2EnAKlSEPDcoTEdzI03Kiyg6iKZXUqnEiqU6AQAIL7yFAE0pfZ0SiN5xgEAiK+RQkntjAIAAKCu3H2Lu7/E3V/s7me5+4eCx9/n7rcEt9/r7i9y93Pc/dXu/vjRtwo0j3t3DuhL9+3W216xTq8/e9WxXzDFkmxan3jzS5UfL+vDt9IEAwAwP9ueG1Jna0qrFrdOPLY+6AqwrZduAIiPSGaX0snExFWyABBHYREA4wCaS1s6qVE6AQCIsdFCSW0UAQAAAKBKCsWy/vibD2v1kjb9waWnznk763o69HuvPkXffuhZ3f5EbxUjBAAsNE88N6T1KzplZhOPhV0BHmckAGIkktmlTEtiIkEGAHFUoAigKbWnUxop0AkAQHyNFIpqzzAOAAAAANVx4x1PaXvvsD505YvmPXbqhlet07qedv3ptx6hQB8AMCfurif2Dem0lZ0veHzV4lZ1ZlLaRhEAYiSS2SU6AQCIu3DkCeMAmks2k1SOLxoAxFguX1KWTgAAAACogqf7R/RPP9yu15+9UpecsWLe28ukkvrzXzpbuwdG9U8/fLIKEQIAFpreobwOjo5PXPkfMjOdtrJTT+yjCADxEcnsUjpFEQCAeAuPcekkiZhmkk1TBAAg3nLjxXlfoQUAAABI0ke+s1XpZELvf8OLqrbNTeuW6apzj9en7tihXftzVdsuAGBheCK40v+0KUUA4WPb9g3J3esdFlATMy4CMLOkmf3MzG6d5rmMmX3ZzLab2T1mtraaQU6VSSVVLLtKZf4iAoinfLGSaM60RLJWK7ay6ZRG8owDABBfdAIAAABANWzde0g/2LpPb33FSVqxqLWq2/6fr12vhJk+8eOnqrpdAED8bdsXFgF0HPbc+hUdOpAbV99Qvt5hATUxm+zS70vaeoTnrpM06O6nSPp7SX8138COJmyPTTcAAHH1fCcAigCaSTudAADE3EihqHY6AQAAAGCePn77U2pPJ3XthWurvu0Vi1p11cbj9fX79+i5g2NV3z4AIL6eeG5I3R0ZLevIHPbcaSsr3QEYCYC4mFF2ycyOl/SLkj59hCVXSropuP01SZeYmc0/vOlRBAAg7iaKAFILtwjAzD5rZr1m9sgRnl9sZt82s4fM7FEz+61ax5TNpJQr0AkAQHzl8iVlM3QCAAAAwNzt7B/Rd7Y8qze/7EQtyaZrso8bXnmySu668Y4dNdk+ACCetu0b0vqVh3cBkKT1wYiAcGQAEHUzzS79g6T/JelIWffVknZLkrsXJR2UtGze0R1BmBTLl7gaE0A8FUoUAUj6nKTLjvL870l6zN3PkfQqSR81s9p8uxDIttAJAEB8ubty44wDAAAAwPx84vbtakkm9NaL1tVsH2u6srpyw3G6+d5d2j9M22YAwLGVy65t+4a1fsWiaZ9f1pFRd0eGIgDExjGzS2Z2uaRed7//aMumecyn2db1ZrbZzDb39fXNIswXyoRFAON0AgAQT+HxLbOAiwDc/Q5JA0dbIqkz6DzTEayt6WX6lU4AJZXLh53iACDy8sWySmVXlnEAAAAAmKNnDozqGw88o6vPW6OezsNbLVfT2191ssaKJf3rnU/XdD8AgHjYPZjT6HjpiJ0AJGn9yg5tYxwAYmIm2aWXS7rCzJ6W9CVJrzGzL05Zs0fSGkkys5SkxZomcePuN7r7Rnff2NPTM+egw6RYeKUsAMRNnk4AM/ExSWdIelbSw5J+391remJoD66OHR2nGwCA+Ak7nbTTCQAAAABz9KmgPf/1rzy55vs6ZXmnLnvRSt3006d1aGy85vsDAERbeIX/aUHb/+mctqJT2/YNcxEYYuGY2SV3f6+7H+/uayVdLemH7v7mKctukXRNcPtNwZqa/Q1JJ4MigCJFAADiKTy+ZZIkYo7itZIelHScpA2SPmZm0/ZyqlYnmrBFNiMBAMRRrlBppkInAAAAAMzFcL6or2zerSs3rNbqJW112efbX3WKhvJFff3+PXXZHwAgusIr/E89ShHA+hWdGh0vac/gaL3CAmpmzpeYmtmHzOyK4O5nJC0zs+2S3i3pPdUI7kgyLRQBAIi3fLGSZA6Pd5jWb0n6hldsl7RT0unTLaxWJ5owMRYmygAgTsICp2yGAjQAAADM3i0PPqtcoaQ3bzqhbvs8+/jFevHxi/Wle3erhtekAQBi4Il9w1q9pE0dmSNf/HDays5gLSMBEH2zyi65++3ufnlw+33ufktwe8zdr3L3U9z9fHffUYtgQ+ngytg8RQAAYioscgo7n2BauyRdIklmtkLSekk1Pf+0B4mxkTydAADEz0i+UuDUTicAAAAAzMHN9+7S6Ss7tWHNkrru9+rzTtAT+4b0s90H6rpfAEC07D0wqhO6skdds2Zp5fm9B+kEgOiLZHYpnJFNJwAAcVUolpVKmBIJa3QoDWNmN0u6S9J6M9tjZteZ2Q1mdkOw5MOSLjSzhyX9l6T/7e79tYypLUiMjY7TCQBA/IwGnQDa0nQCAAAAwOw88sxBPfzMQV193hqZ1fe7jCs2HKdsOqkv3burrvsFAERL71BePZ2Zo67pak8rYVLvoXydogJqJ5KX+UwUAZS4EhNAPBWK5Ylj3ULl7r92jOeflfT/1CkcSVJ7mk4AAOJrJCgCWAidAMzss5Iul9Tr7mdN87xJ+kdJr5eUk3Stuz8w6flFkrZK+qa7v6M+UQMAADSvL923S5lUQr/0kuPrvu+OTEpvePFxuuWhZ/Wnl5+pztaWuscAAGhu7q7eoTEtP0YRQDJh6u7IqHdorE6RAbUTyQxTJkiM5cfpBAAgnvLF8sSxDs0jGyTGcgU6AQCIn/DYls0siE4An5N02VGef52kU4Of6yV9YsrzH5b045pEBgAAEDG5QlHf+tmzev3Zq7Q425gE/NXnr9HoeEm3PPRsQ/YPRJ2ZfdbMes3skUmPdZnZbWb2ZPB7aSNjBOZjOF/U2HhZyxcdvQhAkpYvyqhviE4AiL5IZpie7wRAEQCAeKITQHNqz9AJAEB85RZQJwB3v0PSwFGWXCnp815xt6QlZrZKkszsXEkrJH2/9pECAAA0v+9s2auhfFFXn7emYTFsWLNEp6/s1Jfu3d2wGICI+5wOL5R+j6T/cvdTVRnF+Z56BwVUS2+Q1F/e2XrMtcs7WyfWA1EWyQxTOhl0AihSBAAgngoligCaUTgnOzdOEQCA+BnJVzoBhMe6BW61pMnfIO+RtNrMEpI+Kun/bUhUAAAATehL9+3Wup52nX9SV8NiMDNdfd4aPfzMQT3yzMGGxQFE1REKpa+UdFNw+yZJb6xrUEAV9R6qJPV7jjEOQJJ6OjIUASAWIplhCltkFygCABBT+WJpouAJzSO8OjaXZxwAgPgJOwFkKQKQJJvmMZf0dknfdfdjXmJmZteb2WYz29zX11f1AAEAAJrB0/0juv/ng/rVjWtkNt0/oernl15yvNKphL7+wJ6GxgHEyAp33ytJwe/lDY4HmLPeoTFJ0vIZFAEsX5TR/uG8SmWvdVhATUUyw5RJVb6YpBMAgLgqFMsTxzo0j7aWYBxAgU4AAOJnpFBUOpVQC0VoUuXK/8n9bI+X9Kykl0l6h5k9LelvJb3FzP5yug24+43uvtHdN/b09NQ6XgAAgIa4dcuzkqQ3nHNcgyORFmdb9KrTevSdLXtJ3AB1RhE0ml3frMYBZFR2af8I3QAQbZH8hi9NJwAAMZcvMg6gGSUSpraWpEYLdAIAED+jhRJdAJ53iyoJfjOzTZIOuvted/8Ndz/B3ddK+p+SPu/uzMUEAAAL1q1b9mrjiUt13JK2RociSbr8nOPUO5TXfU9P7WoOYA72mdkqSQp+9x5pIUXQaHZ9Q3mlUwktaksdc204MiAcIQBEVSQzTBQBAIi7AkUATas9k6QTAIBYGsmXJsaexJ2Z3SzpLknrzWyPmV1nZjeY2Q3Bku9K2iFpu6RPqTIGAAAAAJM8uW9Ijz83pMtfvKrRoUy45PTlam1JTHQoADAvt0i6Jrh9jaRvNTAWYF56h/Lq6cjMaHRNT9AtIOweAERVJL/lSyZMqYSpUCIJAyCe8sWyOlsjeYiOvWw6pVyeTgAA4idXKC6YTgDu/mvHeN4l/d4x1nxO0ueqFxUAAEC0fHvLXplJrz+7eYoA2jMpXXL6Cn3v4ef0gTe8SClGXQEzEhRKv0pSt5ntkfR+SX8p6Stmdp2kXZKualyEwPz0Do1p+aLMjNYuDzsBDI3VMiSg5iKbYUqnEsqP0wkAQDwVimVl6ATQlLLppHJ0AgAQQ7lCSdlMZD8eAAAAoI7cXbdueVYXnNSl5YuOPV+5ni5/8Sp95+G9unvHgC46tbvR4QCRcJRC6UvqGghQI31DeZ3U3T6jteE4ADoBIOoim2FKpxIqlCgCABBPhRLjAJoVRQAA4ipXKCrbsjA6AQAAAGB+tu4d0o6+Eb3hnOMaHcphXn36crWnk4wEAABM6B3KTyT3j6W1JalFrSn1UgSAiItshimdTKhQpAgAQDzliyWlaVnXlNozKY0JP9udAAAgAElEQVQUGAcAIH5G8iW1ZygCAAAAwLHduuVZJROm153VPKMAQq0tSf3CmSv0vUee4/tjAIDyxZIO5Ma1vHPmnWuWL2pV7yGKABBtkc0wZVooAgAQX5VxACRimlE2ndQonQAAxNDoeEnZNOMAAACoNzNrNbN7zewhM3vUzD44zZqMmX3ZzLab2T1mtrb+kQIVlVEAe3XhycvU1Z5udDjTuvzFx+ng6Lju3N7f6FAAAA3WP1yQJC2fYSeAcG3fMEUAiLbIFgGkkwnlKQIAEFOFIuMAmlU2TScAAPE0ki8qm6YADQCABshLeo27nyNpg6TLzGzTlDXXSRp091Mk/b2kv6pzjMCER589pF0DOb3hxc03CiD0itO6tag1pVu37G10KACABus9NCZJMx4HEK7tHRqrVUhAXUQ2w5ROJSkCABBbFAE0r2w6qVyeTgAA4idXoBMAAACN4BXDwd2W4MenLLtS0k3B7a9JusTMrE4hAi/w/cf2KWHSpWeuaHQoR5RJJfWa05frh4/vU6k89a8TAGAh6R2qXNE/q3EAnRn1HsrLnXMIoiuyGaZMKqFCiSIAAPGUL5aVoQigKbVn6AQAIH7cXSOFotozdAIAAKARzCxpZg9K6pV0m7vfM2XJakm7Jcndi5IOSlpW3yiBih88tk8bT+xq2lEAoUvPXKHB3Lge2DXY6FAAAA00UQSwaDbjAFqVL5Z1aIzvgRFdkc0wpVMJ5ce5EhNA/JTLrmLZ6QTQpNpakhobL3MlAYBYyRfLcpfaGAcAAEBDuHvJ3TdIOl7S+WZ21pQl0131f9iHEjO73sw2m9nmvr6+WoSKBe6ZA6N6bO8hXXrm8kaHckwXn9ajlqTpB4/ta3QoAIAG6hvKy0xaNovitbBgoC8oIACiKLIZJjoBAIir8NhGEUBzCq+SHaUQDUCMjOQrle3tjAMAAKCh3P2ApNslXTblqT2S1kiSmaUkLZY0MM3rb3T3je6+saenp8bRYiH6r62VhPqlZzTvKIDQotYWbVq3TD/YShEAACxkfUNjWtaeVio58+/bezoqRQC9Q2O1CguouchmmNLJhApFigAAxE8+OLalZ/GPEtRPOC87l6cVFID4yBUqhU1ZOgEAAFB3ZtZjZkuC222SLpX0+JRlt0i6Jrj9Jkk/dIbUogFue2yf1nW3a11PR6NDmZFLz1ihp/pGtKNvuNGhAAAapPdQXj2drbN6DZ0AEAeRzTBlWigCABBP+WIlEZNpIRHTjMJOAGHCDADiIDymtWfoBAAAQAOskvQjM9si6T5Jt7n7rWb2ITO7IljzGUnLzGy7pHdLek+DYsUCNjQ2rrt37NelZzZ/F4DQJWdUxhb819beBkcCAGiU3qG8lndmZvWasGig9xBFAIiuyH7Ll04mJq6WBYA4CQucMnQCaEptLZVT50iBTgAA4iM8prXRCQAAgLpz9y2SXjLN4++bdHtM0lX1jAuY6o5t/RoveSRGAYSOX5rVGasW6bat+/S2i9c1OhwAQAP0DeW1fmXnrF6zqDWldCqhvmGKABBdx8wwmVmrmd1rZg+Z2aNm9sFp1lxrZn1m9mDw89bahPu8dIpOAADiKTy2pVMUATQjOgEAiKNcPugEkI5sjTAAAABq7L+27tPSbIteesKSRocyK79wxnJtfnpAgyOFRocCAKizctnVPzz7TgBmpuWdGfUeGqtRZEDtzSTDlJf0Gnc/R9IGSZeZ2aZp1n3Z3TcEP5+uapTTSKcSKpQoAgAQP3mKAJpaNkiQUQQAIE5yQSeALJ0AAAAAMI1iqawfPtGrV5++XKmIdS689MwVKrv0oycYCQAAC81ArqBi2WddBCCpUgQwRCcARNcx/8XmFcPB3Zbgx2sa1QxkUkk6AQCIpYlxABQBNKUwQZbLMw4AQHyEhU0UAQAAAGA69/98UAdy45EaBRA667jFWt6Z0Q+27mt0KACAOus9VEniL1/UOuvXLu9spQgAkTajDJOZJc3sQUm9km5z93umWfbLZrbFzL5mZmuqGuU00qmE8kWuwgQQP2GXEzoBNKewVfYInQAAxMhI0AmgPcM4AAAAABzuh0/0KpUwveLU7kaHMmuJhOmSM5brv7f1a5zOsgCwoPQNV5L4PXPoBNDTmVEfRQCIsBllmNy95O4bJB0v6XwzO2vKkm9LWuvuL5b0A0k3TbcdM7vezDab2ea+vr75xK10MqHxkqtcbnhTAgCoqrATQDpi7fUWimymcpXsaIFOAADiY5ROAAAAADiKO7b169wTl6qztaXRoczJxaf2aChf1IO7DzQ6FABAHfUeGpOkOY8DODg6rrFxLgZDNM0qw+TuByTdLumyKY/vd/ewHOZTks49wutvdPeN7r6xp6dnDuE+L9NSCb1A9SaAmAm7nGRaSMQ0ozBBRicAAHEykg+LAOgEAAAAgBfqPTSmrXsP6ZXr5/d9biNdeEq3kgnTj5+Y34VpAIBoCdv5L++cwziARZXCAboBIKqOWQRgZj1mtiS43SbpUkmPT1mzatLdKyRtrWaQ0wmvkM0XKQIAEC90AmhuramkzKRcnk4AAOIjVygqk0oombBGhwIAAIAmc8eT/ZKkV54W3SKAxW0teukJS3THkxQBAMBC0jeUV2cmpbY5dD4MCwfCkQJA1Mwkw7RK0o/MbIuk+yTd5u63mtmHzOyKYM27zOxRM3tI0rskXVubcJ+XCWZlFygCABAzYXFTOkURQDNKJEzZliSdAADEykihqPYMXQAAAABwuB9v61N3R0ZnrFzU6FDm5eJTe7Rlz0H1k8wBgAWjbyivnjmMApA08breQ5w3EE3H/KbP3bdIesk0j79v0u33SnpvdUM7ujA5xjgAAHETFgFkKAJoWtlMSjmKAADESK5Qmhh3AgAAAIRKZddPnuzTq09frkTEu0a9cn2PPnrbNv3kyX698SWrGx0OAKAOeofG5lwEsLwzHAcwVs2QgLqJbIYpk6p8SUknAABxU6AIoOll00nlCowDABAfuTxFAAAAADjcw88c1GBuPNKjAEJnHbdYXe1p3bGNkQAAsFDMpxPAso6MEib1DtEJANEU2QxT2AkgX+RKTADxUmAcQNPLplMayXP+ARAfI4WismnGAQAAAOCF7tjWJzPpolO6Gx3KvCUSplec2q07nuxTueyNDgcAUAf7Rwrq7phbEUAyYVqSTWtgpFDlqID6iGyGKZ0MxgHQCQBAzIRjTigCaF7t6aRGx+kEACA+RgsltWfoBAAAAIAX+vG2Pp29erGWzTGB0mwuPrVH/cMFPbb3UKNDAQDU2HiprKGxopZm03PextJsiwZzFAEgmiKbYcq0UAQAIJ7y40ERQDKyh+jYa0sn6QQAIFZGCiW1tdAJAAAAAM87mBvXz3YNxmIUQOgVp1U6GvyYkQAAEHth8r6rvWXO2+hqpxMAoiuyGSY6AQCIq0KppGTClKIIoGm1p1PKFegEACA+coUinQAAAADwAnc+1a+ySxfHqAhgeWerzly1SHdQBAAAsTc4Mi5JWto+n04A6YntAFET2QxT2CY7TxEAgJgpFMt0AWhy2UxSuQKdAADER65QUjZNJwAAAAA877+f7FNnJqUNa5Y0OpSquvi0Ht3/80EN5ynuB4A4C6/g75rHOICu9rQGGAeAiIpslokiAABxlS+WJ45xaE6VTgAUAQCIj1y+qPY0nQAAAADwvJ9s79cF65apJWYXKlx0SreKZdd9OwcaHQoAoIbCcQDz6gTQntbgSEHuXq2wgLqJ7L/gMqnKl5SFEkUAAOKlUCwrQxFAU8umkxrhigEAMVEuu3LjJWUpAgAAAEBg90BOuwdGddEpyxodStVtXLtU6VRCP9ne3+hQAAA1NNEJYB5FAF3ZtIpl1xDfBSOCIptlChNk+XGuxAQQLwU6ATS9bDqlfLGsUpkKUADRN1YsyV3KZhgHAAAAgIo7gwT5y0/pbnAk1dfaktTGE5dOvEcAQDwNBkUAS7Itc95G2EUg3BYQJZHNMoUJMjoBAIibfIkigGbXnqlcLZsrUAEKIPrC8SaMAwAAAEDozqf2a3lnRqcs72h0KDXx8lO69fhzQ+ofzjc6FABAjQzkCurIpCY6i89FV3ulgGCAIgBEUGSzTOlgFlWhSBEAgHjJj5cnjnELmZl91sx6zeyRo6x5lZk9aGaPmtmP6xVbWzosAqAbDYDoy+Urx7K2NJ0AAAAAILm77nqqXxeevExm1uhwauLCkytjDu56an+DIwEA1MrgSEFL2+feBUCSlmaDTgA5igAQPZHNMmVaKAIAEE+FUlmZFq7GlPQ5SZcd6UkzWyLp45KucPcXSbqqTnGpPUiUjTALCkAMjARdTegEAAAAAEl6Yt+Q+ocLujCGowBCZ69erM5MSj99ipEAABBXA7lxdQVJ/LnqCsYBDIyMVyMkoK4iWwQQXiWbpwgAQMwUiiVl6AQgd79D0sBRlvy6pG+4+65gfW9dApOUpRMAgBgJj2XZDJ0AAAAAIP3kyUpi/OUxLgJIJRO6YN0y3bmdTgAAEFeVTgDzKwIIXz/IOABEUGSzTKlkQgmjEwCA+CkUy0qnInt4rqfTJC01s9vN7H4ze0u9dpwNOgFQBAAgDnJBJ4AsnQAAAAAg6adP7ddJ3e1avaSt0aHU1MtPWaZdAzntHsg1OhQAQA0MjBTm3QmgM5NSKmEaYBwAIijSWaZMKqlCiSIAAPGSL5aVoQhgJlKSzpX0i5JeK+lPzey06Raa2fVmttnMNvf19c17x9lMJVEWttAGgCgbyQedACgCAAAAWPDGS2Xds2O/Ljx5WaNDqbmw0wEjAQAgng7k5t8JwMy0tD2tAxQBIIIinWVKpxJ0AgAQO3QCmLE9kv7D3UfcvV/SHZLOmW6hu9/o7hvdfWNPT8+8d9wedAIYpRMAgBgYHa8UNIXHNgAAACxcW/Yc0EihFOtRAKFTl3eopzPDSAAAiKGx8ZJGCiV1zbMIQJK6smkNMA4AERTpLFM6lVC+SAIGQLwUShQBzNC3JL3CzFJmlpV0gaSt9dhxeLXsSJ5OAACib6ITQIZOAAAAAAvdndv3y0x62br4dwIwM7385GX66VP75e6NDgdoWmb2h2b2qJk9YmY3m1lro2MCjuVAblyStHSe4wAkaWl7iwZHxue9HaDeIp1lSicTytMJAEDM5MfLSicjfXiuCjO7WdJdktab2R4zu87MbjCzGyTJ3bdK+g9JWyTdK+nT7v5IPWILiwBydAIAEAO5YLRJlk4AAAA0hJmtMbMfmdnWIMny+9OseZWZHTSzB4Of9zUiVsTfndv7deaqRfNunxwVF57Srf7hvLbtG250KEBTMrPVkt4laaO7nyUpKenqxkYFHFt45X5Xe8u8t9XVntYA4wAQQZH+pi/TwjgAAPFTKJWVaaEIwN1/bQZr/kbS39QhnBdoz1ROnyMFOgEAiL6woKmthU4AAAA0SFHSH7n7A2bWKel+M7vN3R+bsu6/3f3yBsSHBWJsvKSf7T6ga152YqNDqZuw48HdO/Zr/crOBkcDNK2UpDYzG5eUlfRsg+MBjmkwSNpXpRNANq1BxgEggiKdZaITAIA4KhTLSidJxDSzTCqhdCox0VYKAKJscKSgztaUkglrdCgAACxI7r7X3R8Ibg+pMuZsdWOjwkL0wK5BFYplbVoAowBCa7qyWr2kTXc9tb/RoQBNyd2fkfS3knZJ2ivpoLt/f+o6M7vezDab2ea+vr56hwkc5vlOAPMvAuhqT2swV1C5zOgYREukiwAyKToBAIifQrGsdCrSh+fYMzP1dGTUP5RvdCgAMG/9wwX1dGYaHUZdmdlnzazXzKYdI2MV/2Rm281si5m9NHh8g5ndFbRq3mJmv1rfyAEAcWdmayW9RNI90zz9MjN7yMy+Z2YvqmtgWBDu3jGghEnnndTV6FDq6mUnL9M9O/eT3AGmYWZLJV0p6SRJx0lqN7M3T13n7je6+0Z339jT01PvMIHDTHQCqEIRwNJsWmWXDo1xQRiiJdJZpkwqSREAgFgpl70yDoAigKbX3ZlR3zBFAACir284r+6OhVUEIOlzki47yvOvk3Rq8HO9pE8Ej+ckvcXdXxS8/h/MbEkN4wQALCBm1iHp65L+wN0PTXn6AUknuvs5kv5Z0r8fYRtciYk5u3vHfp21erEWtc5/fnKUbFq3TIO5cW3rHWp0KEAzulTSTnfvc/dxSd+QdGGDYwKOKewEsKRt/ue0sJvAACMBEDGRzjKlUwkVShQBAIiP8JhGJ4Dm19ORVv8w//ADEH39w3n1LLAiAHe/Q9LAUZZcKenzXnG3pCVmtsrdt7n7k8E2npXUK4nLXAAA82ZmLaoUAPybu39j6vPufsjdh4Pb35XUYmbd06zjSkzMydh4SQ/uOrCgRgGENq2rdD5gJAAwrV2SNplZ1sxM0iWqjK0BmtrgSEGL21qUSs7/e/awm0DYXQCIimP+329mrWZ2b9Bu7FEz++A0azJm9uWgXeY9QeuymkunEsoXS/XYFQDURVgEQCeA5tfdkVE/nQAAxED/UF7dHfNvjxczqyXtnnR/j6bMZjaz8yWlJT013Qa4EhMAMFNBUuUzkra6+98dYc3KYF14DkpIImOJqnng54MqlMp62QIsAjh+aVZrutp09w7+SgFTufs9kr6mSkeah1U5/9zY0KCAGRjIjU9cwT9fXdmwEwDjABAtqRmsyUt6jbsPB1XJPzGz7wVXxISukzTo7qeY2dWS/kpSzedjppMJxgEAiJX8OJ0AoqK7I6OBkYLKZVciYY0OBwDmJF8s6dBYcSGOAziW6Q7sE0NizWyVpC9Iusbdp/1A4u43KvhybOPGjQyYBQAczcsl/aakh83sweCxP5Z0giS5+yclvUnS75pZUdKopKvdnfMLqubuHfuVMGnj2qWNDqUhNp20TLdt3cdnfGAa7v5+Se9vdBzAbAyOFLQ0W53xNkvbWya2CUTJMYsAgg8Uw8HdluBn6oeMKyV9ILj9NUkfMzOr9YeRTAtFAADihU4A0dHdkVap7BrMFbSM5BmAiNofjDXp7uQ4NsUeSWsm3T9e0rOSZGaLJH1H0p9MKYwGAGBO3P0nmr4AbfKaj0n6WH0iwkJ01479Onv1YnW2VidhEjWb1i3TV+/fo8efG9KZxy1qdDgAgHkaGCnouCWtVdlW2FFggHEAiJgZZZnMLBlUIvdKui1oATPZRLtMdy9KOiip5r2j6AQAIG7CYxqdAJpfmDDrH+YffwCiKxxrQieAw9wi6S1WsUnSQXffa2ZpSd+U9Hl3/2pjQwQAAKiO0UJJD+4+oE0LcBRAaNPJlffOSAAAiIfBXEFLs9UZB9DWklQmlaATACJnRlkmdy+5+wZVroA538zOmrLkqO0yJxZVeS5mOpVQniIAADEyUQSQTDY4EhxLmDALE2gAEEXPFwFU54NxVJjZzZLukrTezPaY2XVmdoOZ3RAs+a6kHZK2S/qUpLcHj/+KpIslXWtmDwY/G+odPwAAQDU9sGtQ4yWfSIQvRKuXtOmErixFAAAQA+6ugZHCxBX882Vm6mpPa4AiAETMMccBTObuB8zsdkmXSXpk0lNhu8w9ZpaStFjSwDSvr+pczHSKTgAA4iVfLEmiE0AUhEUAfUMUAQCIrvAYttA6Abj7rx3jeZf0e9M8/kVJX6xVXAAAAI1w11P7lUyYNp64tNGhNNSmdV36z0f3qVx2JRJHndABAGhio+Ml5YtlLalSJwBJWpJNa5BxAIiYY2aZzKzHzJYEt9skXSrp8SnLbpF0TXD7TZJ+GHxxVlOZVFL5EkUAAOIjLGzKUATQ9HroBAAgBsKRJj2dC6sIAAAAAM+7e8d+nbV6sTpbWxodSkNtWrdMB0fHtfW5Q40OBQAwD+EV+13t1TuvdbW30AkAkTOTLNMqST8ysy2S7pN0m7vfamYfMrMrgjWfkbTMzLZLerek99Qm3BcKOwHUod4AAOpiYhwARQBNb1FbSulkQn0UAQCIsL6hvDoyKbW2MIYGAABgIRotlPTQngPatK6r0aE03KZ1lXEId+84rMEtACBCBkfGJUlLq9gJYGk2rcHceNW2B9TDMccBuPsWSS+Z5vH3Tbo9Jumq6oZ2bOGVsoVSWZkUX1wCiL48RQCRYWZa1pFW/xAVoACiq384r+6O6n0oBgAAQLT8bNegxkuuTScta3QoDXfckjat6WrTvTv367qLTmp0OACAORrIhZ0Aqvd9R1d7mk4AiJxIZ5kmigCKjAQAEA95xgFESndHhnEAACKtUgTAKAAAAICF6u6dA0qYdO7apY0OpSlccNIy3btzQOUynWcBIKoGg2T90ioWASzNpnVwdFxFRpQjQiKdZUpTBAAgZgoligCipLsjTREAgEjrHy5QBAAAALCA3btzv848bpEWtVZvbnKUnX9SlwZz49reN9zoUAAAcxResd9VxXEAYVeBA6OMBEB0RDrLlE5Wws9TBAAgJsKipnSSESdR0NNJJwAA0dY/nFdPJ0UAAAAAC1G+WNLPdh3QBYwCmBCORbhnx/4GRwIAmKvBXEEJkxa1Va/ALewqMMhIAERItIsA6AQAIGbyxZKk549vaG7dHRntHy7QJhBAJI2XyjqQG6cTAAAAwAL10O6DyhfLOv+krkaH0jTWdLVp5aJW3b1zoNGhAADmaGCkoCXZtJIJq9o2w64CAxQBIEIinWXKpCpXyhaYwQEgJsKiJsYBREN3R0bFsusgbaAARND+4coH1+7O6rXHAwAAQHTcu7Nytfv5aykCCJmZLljXpXt3Dsidgn8AiKLBXEFLs9Udc7O0vWVi20BURDrLRCcAAHEzMQ6AIoBI6A5aaDMSAEAUhccuOgEAAAAsTPfsHNDpKzsnWhyj4oKTlqlvKK+d/SONDgUAMAeDI+PqqvK5LdzeYI6LwRAdkc4yhUmysH02AEQdRQDR0t1R+cdfH0UAACKojyIAAACABWu8VNb9Px9kFMA0wj+TexkJAACRVOkEUN0igKWMA0AERTrLlJkoAqATAIB4yBfLSpiUquK8ItROT5A46xuiCABA9ITHrh6KAAAAABacR545qFyhpAtOWtboUJrOyT3t6u7I6B6KAAAgkgZGClXvBNDaklQ2ndQgRQCIkEgXATAOAEDcFEplpVMJmVEEEAXh1bP9w/zjD0D0TIwD6KT9KwAAwEITXuVOJ4DDmZkuOKmLTgAAEEHuXukEUINRN0uzaQ3k+B4Y0RHtIoAkRQAA4qVQLE8c29D8Fre1KJWwiUQaAERJ/1BB2XRS2XSq0aEAAACgzu7ZOaB1Pe3q6aQr1HTOP6lLzxwY1e6BXKNDAQDMwnC+qPGSa2m2perb7mpPMw4AkRLpTBPjAADETb5YUjqVbHQYmKFEwrSsI61+xgEAiKD+4fxERxMAAAAsHKWy676dA4wCOIoL1lU6JDASAACiZXBkXFLlqv1qW5Jt0WBuvOrbBWol4kUAlUQZnQAAxEW+WJ4ocEI0dHdk6AQAIJIqRQCMAgAAAFhotu49pKF8URcwCuCITlveqSXZFt2zY3+jQwEAzELYrr+rBuMAutrTGqQTACIk0pmmdJAoK5QoAgAQDwWKACKnUgTAP/4ARA+dAAAAABamcNb9+RQBHFEiYTpvbZfue5pOAAAQJYNBEcDSGhQBLM2mJ7YPREGkM01hEUB+vNTgSACgOgrF8sSxDdFAJwAAUdU/XFA3M2ABAAAWnHt3Duj4pW06bklbo0Npauev7dLT+3Pad2is0aEAAGYovFK/FuMAlmbTGhorapwLkxERkc400QkAQNzkKQKInO7OtPYPF+TujQ4FAGasWCprMFegEwAAAMAC4+667+kBugDMQPhnFHZOAAA0v8HcuCSpqwZFAF3tLZKkA8E+gGYX6UxT2DK7UKQIAEA8MA4geno6MiqUyjo0Wmx0KAAwYwMjBblLPR3V/1AMAACA5vVU34j2jxR0AUUAx/Si4xbp/2fvzsPjrM974X/v2TQa7RpJlixZmze84A3jBQgQIARoAmTrgTQLBOLSpFnO6elpcs55w1WavH1PT66eNkkbDgmE0KaQhKYEEggEQliMF2y8IC940WLLkixpRuuMpNHM3O8fMyMcY1uyrZnf88x8P9el65I0D9K36SV5NM/9+94+j5NDAERENjIQisAhQJHXNetfuzQ5WMCVAGQXtr7T5HIIRDgEQETZIxJjE4DdVCartPu4EoCIbCT1O6uS6wCIiIiIckrqhva6Jr/hJNbncjpwWUMZhwCIiGwkGI6gzOeBwyGz/rXLCxJDAMEQhwDIHmx9p0lE4HE6MMEhACLKEhPRGDxOW/9qzjmpKu1+DgEQkY30jyb+YOU6ACIiIqLcsr0tgMqiPDT6faaj2ML6pnK8c3Jkasc0ERFZ22A4grKC9LQelvpS6wD4bwLZg+3vNOW5OARARNkjsQ7AaToGnYfUDbS+EQ4BEJF9pH5ncQiAiIiIKHeoKra1BbGuqRwis39CMhulGhPebGcbABGRHQRDEZQlb9bPtnebACbT8vWJZpvthwA8LiciMQ4BEFF2iES5DsBuKpL7tNkEQER2kvqdVcF1AEREREQ5o3NgDN1D41jfVG46im2sqCuBx+XgSgAiIpsYDE+izJeeJoDU1x1gEwDZhO3vNOW5HIiwCYCIsgSHAOynzOeB0yEcAiAiW+kfmYDX7UCBh+0zREREponIPBF5WUQOiMg+EfnKGa4REfmOiBwRkb0issZEVrK31I3sdRwCmDGv24lV80qxnU0ARES2kGgCSM8QgNftRL7byRUxZBu2v9Pk4ToAIsoiExwCsB2HQ1Be4EH/CJ/8EZF99I9OoKIwjzWwRERE1hAF8BequgTABgBfFJGlp11zM4CFybdNAL6f2YiUDba3BVGS78aiqiLTUWxlfVM5Wk4MYXQiajoKERGdg6ommgAK0jMEACRWAgyEuQ6A7MH2d5oSTQAx04vPfwYAACAASURBVDGIiGZFJBpHHocAbGdOcR66h8dNxyAimrHuoXHMKfaajkFEREQAVLVbVd9Kvj8C4ACA2tMuuw3AY5qwFUCpiNRkOCrZ3Pb2IC5vLIfDwUHQ87GuqRxxBXZ2DJiOQkRE5xCKxBCJxVFe4E7b9yj1ubkOgGxj2jtNM6wku1ZEhkRkd/LtG+mJ+14ergMgoiwyEWMTgB01+AtwLBAyHYOIaMaOBcNo8PtMxyAiIqLTiEgjgNUAtp32UC2A46d83In3DgoQnVXv8Dja+kNYz1UA521NfRmcDsH2toDpKEREdA6pmv7SNK0DABJNAEGuAyCbcM3gmlQl2VsiUgRgp4j8VlX3n3bda6r6odmPeG4eJ9cBEFF2UNVEE4CTQwB20+j34fmWHkzG4nDz/39EZHHjkzF0D42j0V9gOgoRERGdQkQKAfw7gK+q6vDpD5/hP9EzfI1NSKwLQH19/axnJPtK7bS/nEMA560gz4XltSXY3hY0HYWIiM4hdUK/PI1DAGU+D44Hw2n7+kSzado7FTOsJDMmz80mACLKDpFY4ndZnttpOAmdr4byAkTjiq7BMdNRiIimdSz5xyqbAIiIiKxDRNxIDAD8RFV/cYZLOgHMO+XjOgBdp1+kqg+p6lpVXVtZWZmesGRL29uC8HmcWD632HQUW1rfVI49x4cwPsm1tEREVpU6oV9WkM4hADebAMg2zuu44jkqyQBgo4jsEZHnRGTZWf77TSKyQ0R29PX1nXfYM/E4HVM3zoiI7Cw10OThSXIAgIg8IiK9ItIyzXWXi0hMRD6eqWynS91I6whwCpSIrC/1u6qBTQBERESWICIC4GEAB1T1789y2dMAPiMJGwAMqWp3xkKS7W1vC+KyhjK4+JrDBVnXWI5ILI7dxwdNRyEiorMYDE8CSNyoT5eyAg+Gx6OI8r4k2cCMn/VNU0n2FoAGVV0J4LsAnjrT10jHNLLHxSYAIsoOU0MALv5BnvQogJvOdYGIOAH8LwDPZyLQ2TRWJG6kdQRCJmMQEc1I6ndVI5sAiIiIrOJKAJ8GcJ2I7E6+3SIi94nIfclrngXQCuAIgB8A+IKhrGRDg+EIDvaMYD1XAVywyxvLIQKuBCAisrDUCf3ytDYBJL724Nhk2r4H0WxxzeSi6SrJTh0KUNVnReSfRaRCVftnL+qZeVxOTHAIgIiywASHAP6Aqr6abKA5ly8h8e/T5WkPdA5VRXnwuh1oZxMAEdlAeyCEYq8LpWnckUdEREQzp6qvA5BprlEAX8xMIso2b7YPAADWNfkNJ7GvEp8bi+cUcQiAcpKIlAL4IYDlABTA51R1i9lURO81EI7AIUCxN71NAAAwEIqgojAvbd+HaDZMe6dpJpVkIlKdvA4isi75dQOzGfRsfG4nwpFoJr4VEVFahSOJvXL5bqfhJPYgIrUAPgLgQQtkQUN5AdcBEJEtdATCUw0mRERERJT9trcF4HE5sKKuxHQUW9vQ7MfOjgFMsgKacs8/AviNql4CYCWAA4bzEJ3RQDiCUp8HDsc5ZysvSmrVwECYTQBkfTM5bjqTSrKPA2gRkT0AvgPgjuSEctqVFrgxEJ5Ehr4dEVHaDIYTdUVlaawryjL/AOCvVDU23YUisklEdojIjr6+vrSEafD7uA6AiGyhIxBGg59DAERERES5YntbEKvmlcLLQwcXZV1TOcYmY2g5MWQ6ClHGiEgxgKuROCgKVY2o6qDZVERnNhCanLpJny6pdQCp1QNEVjbtOoAZVpJ9D8D3ZivU+SjzeRCJxjE2GYPPM6PtBkRElpSaHkz3E5UsshbAE8kimgoAt4hIVFWfOv1CVX0IwEMAsHbt2rRMjTVWFOD3h/oQj2tap02JiC5GJBpH50AYt62aazoKEREREWXA6EQULV3D+MK1801Hsb3LG8sBANvaglhdX2Y4DVHGNAPoA/AjEVkJYCeAr6gqT8KQ5QyEI1M36dOlPHmAL3Wgj8jKbL94mtUbRJQtBlJNANzRPCOq2qSqjaraCOBJAF840wBAptSX+xCJxtEzPG4qAhHRtE4MjiGuid9ZRERERJT9dnYMIBZXrG/ym45ie5VFeWiuLMD2tqDpKESZ5AKwBsD3VXU1gBCAr51+USZaOImmEwxF0t6yO9UEwCEAsoEsGAJI/MANsHqDiGwu9XuM6wASRORxAFsALBaRThG557RVNJbSmKzW7giEDSchIjq71NqSxgquAyAiIiLKBdvbAnA5BGsaSk1HyQrrm8rxZnsQsThX01LO6ATQqarbkh8/icRQwB9Q1YdUda2qrq2srMxoQKKUwXD61wHke5zwuh0Y5MFksgHb9+enbpYNcOqGiGxuIDwJt1NQ4OGOPgBQ1TvP49q70hhlRhr8iVO1HYEQNs7nCQsisqbUoFLqdxYRERERZbftbUEsry3hGtVZsq6pHI9vP46DPcNYNrfEdByitFPVHhE5LiKLVfUdANcD2G86F9HpVBXBcPqbAIDE4eQgDyaTDWRNEwB/4IjI7gZCiZ1FyR33ZDNzS/Phdgra2QRARBbWHgjB53GisjDPdBQiIiIiSrPxyRj2HB/C+qZy01GyRmqtAlcCUI75EoCfiMheAKsA/L+G8xC9RzgSQyQaR3kGVu2W+TxsJydbyIIhgES1B6s3iMjuBsKRqcEmsh+nQzCvzDdVtU1EZEUdgTDqy30cOCMiIiLKAbuPDyISi2MdhwBmzdzSfNSV5XMIgHKKqu5OVv2vUNXbVXXAdCai06XawjPx+npZgZvt5GQLth8CKMlPDAGwCYCI7G4wPInSNO8sovRq8PumqraJiKyoIxBCo7/AdAzjROQREekVkZazPC4i8h0ROSIie0VkzSmPfVZEDiffPpu51ERERETnZ1trECLA2gYOAcymdU3l2NYWhKqajkJEREkDocRB4UytAxjgwWSyAdsPAbicDpTkuzHIqRsisrlgOILyDDxJofRp8BegIxDiCwFEZEmxuOJ4cAwNFT7TUazgUQA3nePxmwEsTL5tAvB9ABCRcgD3A1gPYB2A+0WkLK1JiYiIiC7QtrYAllQXo4QHDmbVhiY/gqEIjvSOmo5CRERJ7zYBpP/fvPICD5sAyBZsPwQAJH6oOXVDRHY3GI6glOsAbK3R70MoEkP/KJ8EEpH1dA+NIRKLswkAgKq+CuBcHa63AXhME7YCKBWRGgAfBPBbVQ0mKzB/i3MPExAREREZMRGN4a1jA1jfzBaA2Zb633QrVwIQEVnG1BBABg7Zlfo8GBqbRDQWT/v3IroYLtMBZkMZp26IyOZUFQPhSZQXcDrfzhqSN9aOBUOoLMoznIaI6A8dS64raShnE8AM1AI4fsrHncnPne3zafPXz+zD/q7hdH4LIiJLWDq3GPd/eJnpGERZY2/nEMYn41jf5DcdJevUl/tQXezF1tYAPr2hwXQcIiLCuyvDyzNwyK7c54YqMDQ2CX8hXwMm68qSJgAOARCRvQ2PRxGLK8rYBGBrDf7EjbX2/rDhJERE79WeGgKoYBPADMgZPqfn+Px7v4DIJhHZISI7+vr6ZjUcERER0XS2tQYAAOub2AQw20QE65vLsa01yHWAREQWMRCehAhQnJ/+Q3aptgE2lJPVZUUTQKnPjXd6RkzHICK6YIPJQSauA7C3ujIfHAJ0BEKmoxARvUdHIASPy4GaYq/pKHbQCWDeKR/XAehKfv7a0z7/+zN9AVV9CMBDALB27doLfnWYp2KJiIjoQmxrC+KS6qKM1CLnovVNfvxydxda+0OYX1loOg4RUc4bCEVQmu+G03Gm2f3ZlTrIx8PJZHVZ0QRQziYAIrK5qboirgOwNY/Lgbml+VOnbYmIrKQ9EMK8snw4MvAHcRZ4GsBnJGEDgCFV7QbwPIAbRaRMRMoA3Jj8HBEREZFlTMbi2NkxwBaANNrQnPjfdltr0HASIiICEjfkM9WyW55qAgjxviRZW1Y0AZQVeBCOxDA+GYPX7TQdh4jovA0mq4PYBGB/jf4CNgEQkSV1BMJo9HMVAACIyONInOivEJFOAPcDcAOAqj4I4FkAtwA4AiAM4O7kY0ER+RsAbya/1AOqyld+iYiIyFLePjGEcCSG9c1+01GyVlNFASqL8rCtLYBPrq83HYeIKOcNhCMZa78p9bmnvieRlWXFEEDqB24wPInqEg4BEJH9pJ4wZGpakdJnQVUhfrbjOGJxzUj9FBHRTEzG4mjtC+GaRZWmo1iCqt45zeMK4ItneewRAI+kIxcRERHRbNjaGgAArGMTQNqICNY3lWNrawCqChH+/U9EZNJAaBJzS/Mz8r2mmgCSB/uIrCpr1gEAnLohIvuaWgfAIQDbWzq3GOFIDO1sAyAiCzl8chSRWBxL5xabjkJEREREabatNYiFVYWoKMwzHSWrrW/24+TwBDq4EpCIyLjEOoDMrNrNdzvhcTm4DoAsLyuGAFL12fyBIyK7GgxPwiFAkTcrClpy2vK5JQCAlhNDhpMQEb2rpSvxO2l5bYnhJERERESUTtFYHDvag1jfzBaAdNuQbFrY1hYwnISIiIKhyNQJ/XQTEZT7PFMH+4isKiuGAFi9QUR2FwxHUObzwMH6eNtbOKcQHqcD+7uGTUchIpqyv2sYPo8TTf4C01GIiIiIKI32dQ0jFIlhfZPfdJSst6CqEP4CD7a1Bk1HISLKaWORGCai8akDw5lQ6nPzniRZXlYMAaQqPoJcB0BENjUYjqA0Q3VFlF5upwOLq4umTt0SEVlBy4khLK0p5rAZERERUZZLnUpnE0D6iQjWN5djW1sQqmo6DhFRzkrdGywvyNzr6+UFHq4oJ8vLiiGA1HTPIKs3iMimBkKTKMvgpCKl17K5xdjXNcwXAYjIEuJxxYHuYSybW2w6ChERERGl2dbWIJorC1BV5DUdJSesb/LjxOAYOgfGTEchIspZqVXhmXx9vYxDAGQDWTEE4HE5UJjnYvUGEdnWQDiCsgztLKL0W1ZbgsHwJLqGxk1HISJCeyCEUCSGZbUlpqMQERERURpFY3FsbwtiYzNXAWTKxvmJ/623HA0YTkJElLuCqSGADL6+Xu7zIDDKIQCytqwYAgBS+zf4A0dE9jQQjkytNiH7S522bTnBlQBEZF5L1zAAsAmAiIiIKMu9fWIIoxPRqRvTlH4LqwpRUejBllYOARARmRIITQAAKgrzMvY9/YUeDI1NYjIWz9j3JDpfWTMEwP0bRGRXqppYB8AmgKyxpLoYDgH2JW+8ERGZtK9rCB6nAwurikxHISIiIqI0St2I3sAmgIwREaxv9uONo/1cCUhEZEjqRL6/MHOvr6cGDga4ppwsbNohABGZJyIvi8gBEdknIl85wzUiIt8RkSMisldE1qQn7tmV+jz8YSMiWwpHYojE4hndWUTple9xYn5lIfaxCYCILGDfiWEsqi6Ex5U1879EREREdAZbjgawaE5hRk9CErCx2Y+TwxNo6w+ZjkJElJP6RifgcTlQlOfK2PesSA4c9I1OZOx7Ep2vmbwSGAXwF6q6BMAGAF8UkaWnXXMzgIXJt00Avj+rKWegzOfGQHgy09+WiOiipVpMuA4guyyvLWETABEZp6rY1zWE5XNLTEchIiKisxCRR0SkV0RazvL4tSIyJCK7k2/fyHRGsr5INI4d7QPYyBaAjLsiuX6BKwGIiMwIjEZQUeCBiGTse/qTA3epFgIiK5p2CEBVu1X1reT7IwAOAKg97bLbADymCVsBlIpIzaynPYcyH9cBEJE9DYQSA0xsAsguy+YWo2d4HP2cBiUig7qGxjEQnsSyucWmoxAREdHZPQrgpmmueU1VVyXfHshAJrKZvZ2DGJuMYeN8DgFkWlNFAeYU52HLUQ4BEBGZEBidmLopnyn+5GrfQIiv/ZJ1nVcnqIg0AlgNYNtpD9UCOH7Kx51476BAWpX5PBgZj2IyFs/ktyUiumhTTQAFHALIJsuSp27ZBkBEJqXWkiyrZRMAERGRVanqqwCCpnOQvW05GoAIsL6JQwCZJiLY2OzH1tYgVNV0HCKinBMIReAvzOxr62wCIDuY8RCAiBQC+HcAX1XV0+9onKlj4z3PeERkk4jsEJEdfX1955d0GuUFiRrtQa4EICKbeXcdAIcAssnS5KnbluQNOCIiE1q6huEQYEk1mwCIiIhsbqOI7BGR50RkmekwZD1vHA1gSXUxDxgYsnG+H/2jEzjSO2o6ChFRzgmMRuAvyGwTQLHXBY/TgX4OAZCFzWgIQETcSAwA/ERVf3GGSzoBzDvl4zoAXadfpKoPqepaVV1bWVl5IXnPqjR584wrAYjIbgZCqSEAt+EkNJtK8t2oL/dhP5sAiMig/V1DmF9ZiHyP03QUIiIiunBvAWhQ1ZUAvgvgqbNdmM4DOGRd45Mx7Dw2wFUABm1srgAAbGnlSgAiokxSVfSPTqAiw00AIgJ/oQcBroIlC5t2CEBEBMDDAA6o6t+f5bKnAXxGEjYAGFLV7lnMOa3UCdrUzTQiIrsYSDaYlORzCCDbLJtbjJYuNgEQkTktJ4axbC5bAIiIiOxMVYdVdTT5/rMA3CJScZZr03YAh6xr17FBRKJxbGzmEIAp88rzUVuajy1HOQRARJRJoUgME9F4xtcBAEgMAfCeJFnYTJoArgTwaQDXicju5NstInKfiNyXvOZZAK0AjgD4AYAvpCfu2ZUl1wEMcB0AEdnMQDiCknw3XM4Zb2ghm1heW4KOQJgDakRkRO/wOHqGx7G8tsR0FCIiIroIIlKdPKQDEVmHxOt5vNNIU7a0BuAQYF1zuekoOUtEsHG+H1tbA4jH37Mll4iI0qR/JHESv6Iws+sAAMBfkId+NgGQhbmmu0BVXwcg01yjAL44W6EuRBnXARCRTQ2EJ7kKIEuta0q8ALO9PYgPLqs2nIaIcs3WtiCAd38XERERkTWJyOMArgVQISKdAO4H4AYAVX0QwMcB/JmIRAGMAbgj+VocEQBg69EAlteWoNjL1xZM2tjsx5M7O3GwZwRL2cZFRJQRgVDiJrzfxBBAoQdHekcz/n2JZmraIQC74BAAEdnVQCiCsoLM1xVR+q2oK4HX7cDW1gCHAIgo47a2BlCU58LSGr4ASUREZGWqeuc0j38PwPcyFIdsJjQRxa7jA/jcVU2mo+S8KxYk1jG8cbSfQwBERBnSP5q4J+g38Pp6RWGiCUBVkSxtIrKUrOmezvc44XU7WLlMRLYzEI5MDTJRdslzObG2oRxbW4OmoxBRDtraGsC6pnKumyEiIiLKYtvbg5iMKd63oNJ0lJxXU5KP+ZUF2Hyk33QUIqKcEUgOAZhZB+DBRDSOUCSW8e9NNBNZ9Ypgmc+DgfCk6RhEROdlMDyJUq4DyFobmstxoHuYQ2pElFEnh8fR2hfChma/6ShERERElEabD/fD43JgbWOZ6SgE4MoFFdjWFkQkGjcdhYgoJwRGE+sAyg01AZyagchqsm8IgDdZiMhmgqEIytkEkLVSN+C2tbENgIgyZ2trAAA4BEBERESU5V4/0o/LG8vgdTtNRyEkhgDCkRh2Hx80HYWIKCcEQhEUe13wuDJ/u9NfmHhNP7WSgMhqsmsIoMCNgTB/2IjIPsYnYxibjKHMwKQiZcaKulLku51TN+SIiDJha2sQRV4Xd5ESERERZbG+kQkc7BnBlQsqTEehpA3NfjgkMZxBRETp1zc6YWQVAPBuE0A/mwDIorJrCMDnwSDXARCRjaR+Z5WxCSBrpWoZOQRARJm0rTWA9U3lcDrEdBQiIiIiSpM3jiZuNF/FIQDLKMl3Y0VdKTZzCICIKCMCBocAUk0AATYBkEVl3RBAkE0ARGQjweQKkzKf23ASSqcNzX4c7BnhyhoiyoiTw+No7Q9xFQARERFRltt8pB8l+W4sm1tiOgqd4qoFFdh9fBAj4zysRkSUboHRyNTN+EwrL0gNAbAJgKwpy4YA3Bgam0QsrqajEBHNyGBycKmUTQBZbUNzOQBgW1vQcBIiygWp5hEOARARERFlL1XF64f7ccV8P9ufLObKBRWIxRXbWvkaABFRugVC5oYA8lxOFHldCPDgF1lUVg0BVBblQZX7N4jIPk6OjAMAKos4BJDNLq0tRb7byZUARJQRW1sDKPK6sKSm2HQUIiIiIkqT9kAYXUPjuJKrACxnTUMpvG4HXudKACKitIrG4hgIR+AvMLMOAAAqC/N4T5IsK6uGAOr9BQCAjkDYcBIiopnpCIQhAtSV+UxHoTTyuBxY21jGIQAiyoitrUGsbyrniTAiIiKiLJa6wXwVhwAsJ8/lxLomPzZzCICyiIg4RWSXiPzKdBailIHwJFSBCkNNAADgL/QgMMomALKmrBoCaEoOAbT3hwwnISKamfb+EOaW5MPrdpqOQmm2odmPgz0jCLIeiojSqGdoHG39Ia4CICIiIspymw/3o7Y0Hw1+HiqwoqsW+HG4dxQ9Q+OmoxDNlq8AOGA6BNGpAqHECXx/obkmAH9B3lQOIqvJqiGAuaVeuByC9gCHAIjIHtoDYTRW8A/2MxGRR0SkV0RazvL4n4jI3uTbGyKyMtMZz0eqovHVQ32GkxBRNnvlUC8AsBaWiIiIKIvF4ootrQFctaACImx/sqLU83GuBKBsICJ1AP4IwA9NZyE6Vf9I4rCVv8BsE0A/mwDIorJqCMDldGBeuY/rAIjINjoCITQkW0zoPR4FcNM5Hm8DcI2qrgDwNwAeykSoC7WitgRVRXl4YX+P6ShElMVe2HcSdWX5uKS6yHQUIiIiIkqTPZ2DGBqbxFULOfhpVUuqi+Ev8OC1wzwIQFnhHwD8NwBx00GITpU6gV9RZLAJoDAPA+EIojH+eJD1ZNUQAAA0+H1sAiAiWxgKT2IgPIlGVvedkaq+CiB4jsffUNWB5IdbAdRlJNgFcjgEH1g6B79/pw/jkzHTcYgoC4UmonjtSD9uXFrNE2FEREREWeyVd/rgEOAqtj9ZlsMhuHpRJV473I94XE3HIbpgIvIhAL2qunOa6zaJyA4R2dHXx+EXyozUCfyKAnNDABWFHqgCA+FJYxmIzibrhgAa/QXoCIShyidXRGRtHcHEwBKbAGbFPQCeO9uDVvlD5MZl1QhHYnjjKOsAiWj2vXqoD5FoHDcum2M6ChERERGl0SuH+rByXinKDNYf0/SuWVSJYCiClq4h01GILsaVAG4VkXYATwC4TkT+9fSLVPUhVV2rqmsrKysznZFyVGB0Ai6HoDjfZSxDRWFiACHVSkBkJVk3BNDg92F0IopAiDs4iMja2pOrSxo5BHBRROT9SAwB/NXZrrHKHyIbm/0oynPhhX0njWUgouz1wv6TKPO5sbahzHQUIiIiIkqTgVAEezsHcc0i3mSzuvctrIBIormByK5U9euqWqeqjQDuAPA7Vf2U4VhEAIDAaAT+Qo/RNkR/ciAvMMp7kmQ9WTcE0FiRuJnW3s+VAERkbanfUw1cB3DBRGQFgB8CuE1VA6bzTMfjcuDaS6rw4oGTiLEOkIhm0WQsjpcOnMT1S+bA5cy6p/hERERElPT6kX7EFRwCsAF/YR4urS3BK4c4BEBElA6B0AT8BlcBAInf9QDQP8omALKerHuFMHWiNnXClojIqtoDIdSUeOF1O01HsSURqQfwCwCfVtVDpvPM1I1L56B/NIJdxwZMRyGiLLK9LYjh8ShuXMpVAERERETZ7JVDfSj1ubGirtR0FJqBaxZV4q1jAxjirmjKAqr6e1X9kOkcRCl9ySYAkyqS37+fTQBkQVk3BFBbmg+nQ9ARYBMAEVlbRyDMFoBzEJHHAWwBsFhEOkXkHhG5T0TuS17yDQB+AP8sIrtFZIexsOfh2sWVcDsFL+znSgAimj0v7OuB1+3A+xbyRBgRERFRtlJVvHKoD1ctqIDTYa76mGbumkWViCuw+Wi/6ShERFknMDqBikKzTQDFXjdcDkGATQBkQS7TAWabx+VAbWk+mwCIyPI6AiHcsIQnNs9GVe+c5vF7AdyboTizpsjrxhXzK/D8vh58/eZLjO6sIqLsoKp4Yf9JXL2wEvketssQERERZasD3SPoG5ngKgAbWTWvFEVeF1491IdbLq0xHYeIKKsERiPwF5htAnA4BOUFHgTYBEAWlHVNAEBivzabAIjIykbGJ9E/GkFDcoUJ5ZYbl81BRyCMw72jpqMQURZoOTGM7qFx3Lis2nQUIiIiIkqj1G55DgHYh8vpwPsWVuCVQ31QVdNxiIiyRjgSxdhkDBVFZpsAAMBfmIdAiE0AZD1ZOQTQ6C9AW3+IT6yIyLI6km0ljVwHkJM+sGQOHAI8s6fLdBQiygK/2tsFl0Nw3SVVpqMQERERURq9cqgXS2qKUVXsNR2FzsM1iyrRPTTOgwBERLModfLedBMAAFQUetDPJgCyoOwcAqgowMh4FAPhSdNRiIjOqD3ZVtJYwSaAXFRV7MVVCyvxi7dOIB7nwBoRXbhoLI5f7DqB919ShXIL/OFLREREROkxOhHFjvYBXL2ownQUOk9XJ5sbXnmnz3ASIqLs0T+aOHlfUWi+CaCCTQBkUdMOAYjIIyLSKyItZ3n8WhEZEpHdybdvzH7M85M6WdvOlQBEZFGpJoAGNgHkrE9cVocTg2PY0howHYWIbOyVQ33oG5nAJy6rMx2FiIiIiNLo9cP9iMaVqwBsqKYkH4vnFOHld3pNRyEiyhpTTQCF5g9E+As8U3mIrGQmTQCPArhpmmteU9VVybcHLj7WxUnt2O7gEAARWVR7fwhVRXnweVymo5AhH1g6B8VeF36+47jpKERkYz/f0Ql/gQfv5yqA8yYiN4nIOyJyRES+dobHG0TkJRHZKyK/F5G6Ux77OxHZJyIHROQ7IiKZTU9ERES55qUDJ1HkdeHyxnLTUegCXLekCtvbghgeZ3MtEdFsSDUBWKEV0V+Yh3AkhnAkajoK0R+YdghAVV8FEMxAllkzrzwfIkB7f9h0WFl3mQAAIABJREFUFCKiM+oIhNHo5yqAXOZ1O3Hrqrl4rqWHLwIQ0QUJhiJ46eBJ3L66Fm5nVm75ShsRcQL4JwA3A1gK4E4RWXraZd8G8JiqrgDwAIC/Tf63VwC4EsAKAMsBXA7gmgxFJyIiohwUjytefqcX1y6u4vM+m7phSRWiceVKACKiWXJyODEEUFXkNZwEmFOcWEmQykRkFbP1rHGjiOwRkedEZNksfc0LludyYm5JPpsAiMiy2gMhrgIgfPyyeZiIxvHrvd2moxCRDf1y9wlMxhQf5yqAC7EOwBFVbVXVCIAnANx22jVLAbyUfP/lUx5XAF4AHgB5ANwATqY9MREREeWs3Z2D6B+N4IYlbH+yq1XzylBe4MFLB/i0kYhoNnQPjaGiMA8el/nhuOqSxCBC99CY4SREf2g2fjreAtCgqisBfBfAU2e7UEQ2icgOEdnR15feqcfGCh/aA2wCICLrCUei6B2ZQGMFmwBy3cq6EiysKuRKACK6ID/f0YnltcVYUlNsOood1QI49ZdvZ/Jzp9oD4GPJ9z8CoEhE/Kq6BYmhgO7k2/OqeuD0b5DJv32IiMjeROQREekVkZazPC7J9TNHkmtq1mQ6I5n10oGTcDoE1y7iEIBdOR2C9y+uwsvv9CEai5uOQ0Rke91D46gpMd8CAAA1JfkAgO7BccNJiP7QRQ8BqOqwqo4m338WgFtEKs5y7UOqulZV11ZWVl7stz6nRn8BmwCIyJI6kgNKXAdAIoJPrK3DW8cGcbRv1HQcIrKRfV1D2N89jE9cNs90FLuSM3xOT/v4vwK4RkR2IVH3fwJAVEQWAFgCoA6JwYHrROTq93yxDP7tQ0REtvcogJvO8fjNABYm3zYB+H4GMpGFvHSgF2sbylDic5uOQhfhhiVVGBqbxM6OAdNRiIhsr8dSQwCJHD3DHAIga7noIQARqRYRSb6/Lvk1Axf7dS9Wo78AA+FJDIW5Z5mIrKW9PzGgxHUABAC3r66F0yH42ZtsAyCimfv5jk54nA7cunKu6Sh21Qng1AmKOgBdp16gql2q+lFVXQ3gfyQ/N4REK8BWVR1NDkM/B2BDZmITEVE2UtVXAQTPccltAB7ThK0ASkWkJjPpyLTOgTAO9ozgeq4CsL2rFlbA7RT87mCv6ShERLbXPTRmmSEAr9uJMp+b6wDIcqYdAhCRxwFsAbBYRDpF5B4RuU9E7kte8nEALSKyB8B3ANyhqqefosm41M211n6erCQia2nlEACdoqrIiw8um4PHtx9DaCJqOg4R2cDQ2CR+vuM4/mhFDcoKPKbj2NWbABaKSJOIeADcAeDpUy8QkQoRSf299HUAjyTfP4ZEQ4BLRNxItAS8Zx0AERHRLJrJGhvKUqkbxtcvmWM4CV2sIq8bG5r9ePHASdNRiIhsLTQRxfB4FNXJGn4rqC7JR88QmwDIWqYdAlDVO1W1RlXdqlqnqg+r6oOq+mDy8e+p6jJVXamqG1T1jfTHnt6y2hIAwN7OIcNJiIj+0N7OQTT4fSjyssaPEu59XzOGx6N4cmen6ShEZAM/ffMYQpEY7rmqyXQU21LVKIA/B/A8Ejfwf6aq+0TkARG5NXnZtQDeEZFDAOYA+Fby808COArgbQB7AOxR1WcymZ+IiHLOTNbYJC4U2SQiO0RkR19fX5pjUSa8eKAXTRUFmF9ZaDoKzYLrL6nC0b7QVEskERGdv+7kzXarNAEAiSxdgxwCIGu56HUAVjW3xIs5xXncsURElqKq2NkxiDX1ZaajkIWsqS/DmvpSPLK5DbG48TIdIrKwyVgcj25ux8ZmP5Ynh17pwqjqs6q6SFXnq+q3kp/7hqo+nXz/SVVdmLzmXlWdSH4+pqp/qqpLVHWpqv4Xk/93EBFRTph2jU2Kqj6kqmtVdW1lZWVGwlH6jE5EsfVoANdfwlUA2SLV6MA2ACKiC5c6cV9toSGA6hIveoY5BEDWkrVDACKCyxrK8NYxDgEQkXV0Doyhf3QCaxo4BEB/6N73NaMjEOYLAUR0Ts+19KBraBz3vo8tAERERDnkaQCfkYQNAIZUtdt0KEq/1w/3IRKLcxVAFplX7sPiOUX825+I6CJ0D40BAOZaaB3A3BIvgqEIxidjpqMQTcnaIQAgcbKyc2AMvZy+ISKLSA0mrakvNZyErObGpXNQV5aPh19rMx2FiCxKVfHD11rRXFGA9y/maTAiIqJsISKPA9gCYLGIdIrIPSJyn4jcl7zkWQCtAI4A+AGALxiKShn2XEsPynxuXN7IgwTZ5MZlc7C9LYjA6ITpKEREtpRaB1BVnGc4ybuqkwMJJ3k/kiwkq4cAVifrttkGQERWsbNjAD6PE4vnFJmOQhbjcjpw95VN2N4exJ7jg6bjEJEF7egYwN7OIXzuqiY4HGdaDUxERER2pKp3qmqNqrpVtU5VH1bVB1X1weTjqqpfTK6wuVRVd5jOTOk3EY3hdwd68YGlc+ByZvVLuDnnpuXViCvw2/1sAyAiuhDdQ+PwF3jgdTtNR5lSk1xNkBpQILKCrH4Guby2GB6nA28d480UIrKGt44NYNW8Uv4BT2f0x2vrUJTnwkOvtZqOQkQW9NCrrSj1ufGxNXWmoxARERFRmm0+0o+RiShuXl5jOgrNsqU1xagv9+G5lh7TUYiIbKlnaAzVyZvuVlE9NQQwZjgJ0buy+i5UnsuJ5bXFeKuDTQBEZF44EsWB7hGsqWeNH51ZkdeNT29swLNvd+Ngz7DpOERkIW93DuG3+0/irisake+xzqQ7EREREaXHc2/3oMjrwhUL/Kaj0CwTEdx8aTXeONqPofCk6ThERLbTPTSOmmT9vlWwCYCsKKuHAADgsoYy7D0xhEg0bjoKEeW4PceHEIsrLmvgEACd3aarm1HoceHvXzhkOgoRWci3X3gHZT437rmqyXQUIiIiIkqzyVgcL+w/iRuWzEGeiwOg2ejm5TWYjClePMCVAERE5ysxBGCtJgCfx4WSfDd6OARAFpL1QwBr6ssQicaxr2vIdBQiynFvHUu0kqyuLzWchKys1OfB569uxgv7T2JvJ9fZEBHwZnsQrxzqw33XzEeR1206DhERERGl2dbWAIbGJnHT8mrTUShNVtaVYG6JlysBiIjOUzgSxdDYpOXWAQCJNgA2AZCVZP8QQPLE7U6uBCAiw97qGEBzZQFKfR7TUcji7r6yEWU+N77NNgCinKeq+Pbz76CyKA+f2dhoOg4RERERZcBzLT3weZy4ZlGl6SiUJiKCDy6vxquH+zA6ETUdh4jINlIn7a3WBAAA1SVeNgGQpWT9EMCcYi9qS/Ox6xhPUxKROaqKXccHcVk9VwHQ9Iq8bvzZtfPx6qE+bG8Lmo5DRAZtPhLAtrYg/vz9C5DvYRUsERERUbaLxRUv7OvB+y+pgtfN53/Z7JZLaxCJxvG7g72moxAR2ca7QwD5hpO8V01JPrqHxkzHIJqS9UMAQKINIFXDTURkQnsgjGAoMtVOQjSdT29oRGVRHr79/DtQVdNxiMiAeFzxv194B7Wl+bhj3TzTcYiIiIgoA95sD6J/NIKbuQog611WX4bKojz8pqXbdBQiItvosnATQE2JF/2jEUxEY6ajEAHIlSGA+lJ0D42ja5ATOERkRmolyRo2AdAM5Xuc+Mr1C7G9PYhfv80XBIhy0X/sOoE9xwfx1RsWIs/FU2BEREREueDZt7uR53Lg/YurTEehNHM4BB9cNgcvH+xDiCsBiIhmpCd50r7agkMAqUy9wxOGkxAl5MQQwGXJk7epm3BERJm2s2MARXkuLKwqNB2FbOTOdfVYWlOMb/36AF8QIMoxw+OT+NvnDmLVvFJ8bE2d6ThERERElAGTsTh+tbcbNyyZg4I8l+k4lAG3rqzF2GQMv91/0nQUIiJb6B4aR5nPbcmVOal2gu5kWwGRaTkxBLCkphgl+W7uVyIiI+JxxcsHe7Fhvh8Oh5iOQzbidAgeuG0ZuofG8U8vHzEdh4gy6B9fPIxAaAIP3LaM/3YQERER5YjXD/cjGIrgtlVzTUehDFnbUIba0nw8tfuE6ShERLbQMzSOmpJ80zHOKJWre4it5GQNOTEE4HY6cMOSOXjxwElEonHTcYgox+zuHETP8Dj3+dEFWdtYjo+uqcUPXmtFa9+o6ThElAGHTo7g0Tfaccfl9VhRV2o6DhERERFlyFO7T6Ak341ruQogZzgcgltXzcVrh/vRP8r6aCKi6XQNjU+duLeaajYBkMXkxBAAANy8vBoj41G8cbTfdBQiyjHPt/TA7RRcv2SO6ShkU1+7+RLkuZx44Ff7oaqm4xBRGqkq7v/lPhTmufCXH1xsOg4RERERZUhoIooX9p3EH62ogceVMy/ZEoCPrK5FLK741Z4u01GIiCyvZ2hs6ma71RTmuVDkdaGHQwBkETnzjPKqhRUo8Djxm5Ye01GIKIeoKp5r6cEV8ytQku82HYdsqqrIi6/esBC/f6cPz+ztNh2HiNLoyZ2d2NIawH/94GKUF3hMxyEiIiKiDHlhfw/GJmO4fVWt6SiUYYvmFGFJTTGe2s0hACKicxmfjGEgPGnZJgAAqCnxch0AWUbODAF43U5ct2QOXth/EtEYVwIQUWbs7x7GsWCYqwDoot11RSNWzSvFN37Zgt4RTpMSZaPuoTE88Mx+rGssx5+sqzcdh4iIiIgy6KldXagtzcfahjLTUciA21fNxe7jg2jvD5mOQkRkWakT9tUl+YaTnF11ST6bAMgycmYIAEisBAiGItjeHjQdhYhyxG9aeuAQ4ANLuQqALo7L6cC3P7ES4UgM/+M/WrgWgCjLqCr+6t/fRjSu+N+fWAGHQ0xHIiIiIqIM6RuZwGuH+3Dbqrl8Hpijbl01FyLAU7tPmI5CRGRZXckT9nMt3AQwt8SLLg4BkEXk1BDAtYsr4XU78DxXAhBRhvympQfrmsrhL8wzHYWywIKqQvzljYvx2/0n+cIAUZb56ZvH8eqhPnz9lkvQ4C8wHYeIiIiIMuhXe7sQV+D21VwFkKtqSvKxocmPX+7u4tA/EdFZvNsEYN0hgOoSL/pHJxCJspGczMupIQCfx4VrFlXiN/t6EI/zyRQRpdeR3lEc7h3FzctrTEehLPK5q5pwWUMZ7v/lPu6XIsoSx4NhfPPXB7Cx2Y9PrW8wHYeIiIiIMuwXb53AkppiLJpTZDoKGfSR1bVo6w9h1/FB01GIiCyp2wZDADUlXqiC61zJEnJqCAAAbl5eg5PDE3wyRURp95uWbgDAB5dVG05C2cTpEHz7EysRjSu+/PguRGOcKiWys0g0jj9/fBcEwN99nGsAiIiIiHJNy4khvH1iCP9pbZ3pKGTYLStq4PM48cT2Y6ajEE0RkXki8rKIHBCRfSLyFdOZKHd1D42hJN8Nn8dlOspZVZfkAwC6BjkEQObl3BDAdUuq4HYKfrW3y3QUIspiqopf7e3GmvpSS08mkj01VRTgbz96Kd5sH8C3XzhkOg4RXYS/fe4A9hwfxN99fAXmlftMxyEiIiKiDHvizWPIcznwkdUcAsh1hXku3LZqLp7Z042R8UnTcYhSogD+QlWXANgA4IsistRwJspRbf0hNFZYe4ViU3LFY3t/yHASohkMAYjIIyLSKyItZ3lcROQ7InJERPaKyJrZjzl7ir1ufHBZNZ7c2YnQRNR0HCLKUtvbgjjYM4KPXzbPdBTKUretqsWd6+rx4CtH8buDJ03HIaIL8JuWbvxoczvuuqIRN1/K1TFEREREuSYcieKpXV34o0trUOJzm45DFnDH5fUYm4zhl7t5gI2sQVW7VfWt5PsjAA4AqDWbinLV0d4Q5ldaewigtiwfHpcDR/pGTUchmlETwKMAbjrH4zcDWJh82wTg+xcfK70+d1UTRsajeHJnp+koRJSlHtnchjKfGx9dw+fElD73f3gpltYU4z//dA86B8Km4xDReegIhPCXP9+LlfNK8d9vWWI6DhEREREZ8Ku93RidiOKOdfWmo5BFrKgrwZKaYjzOlQBkQSLSCGA1gG1mk1AuGp2Iomd4HPMrC01HOSenQ9BcUYCjvRwCIPOmHQJQ1VcBBM9xyW0AHtOErQBKRcTSR5nW1JdhdX0pfrS5DfG4mo5DRFnmWCCMF/afxCfX18PrdpqOQ1nM63bin/9kDeJxxabHdrLhhsgmhscnce+Pd8DhEHzvztXwuHJuQxcRERERAXh8+zHMryzA5Y1lpqOQRYgIPrluHvZ1DePtziHTcYimiEghgH8H8FVVHT7D45tEZIeI7Ojr68t8QMp6qZvqVh8CABIZj7IJgCxgNl5xrAVw/JSPO3GWOhgr/UPwuSub0B4I43cHe43mIKLs8+gb7XCK4DMbG01HsbVsW0eTLo0VBfjuJ1fjYM8wvvrT3RxuI7K4aCyOL/3bLrT1h/D9P1mDeeU+05GIiIiIyIB3ekaw69gg7lxXDxExHYcs5LbVtfC6Hfg3tgGQRYiIG4kBgJ+o6i/OdI2qPqSqa1V1bWVlZWYDUk5I3VRfUGWDIYCqQhwLhjERjZmOQjluNoYAzvQs9Yx3IKz0D8FNy6tRU+LFI5vbjOYgouwyMj6Jn+04jg+tqMGcYq/pOHb3KLJsHU26XLu4Ct/40FL8dv9J/K/nD5qOQ0Tn8M1fH8Arh/rwN7cvxxULKkzHISIiIiJDHt9+DB6nAx9dU2c6CllMsdeND62Yi6d3n2DjHxkniSmlhwEcUNW/N52HctfRvlG4HIIGv/UPU8yvLEBcgY4A17eSWbMxBNAJYN4pH9cB6JqFr5tWbqcDn72iEW8cDWB/13vaa4iILsjPdnRidCKKz13VZDqK7WXjOpp0+uwVjfj0hgb831da8QRPCxBZ0mNb2vHoG+2496om3Mm9r0REREQ5KzQRxS/e6sQHl1ejvMBjOg5Z0J3r5iEUieE/dp0wHYXoSgCfBnCdiOxOvt1iOhTlnqO9IdT7fXA7rb9SMbWyILXCgMiU2fhpeRrAZ5K1zBsADKlq9yx83bS78/J65LudePh1tgEQ0cWbjMXx6BttuLyxDCvqSk3HyQUzXkeTC0QE9394Ka5eVIn//h9v4zctPaYjEdEpnt7Thfuf3ocbllTh67csMR2HiIiIiAx6cmcnhsejuOuKRtNRyKLW1Jfh0toSPLK5jWv/yChVfV1VRVVXqOqq5NuzpnNR7jnaNzp1c93qmisLALy7woDIlGmHAETkcQBbACwWkU4RuUdE7hOR+5KXPAugFcARAD8A8IW0pZ1lJT43Prm+Hv+xqxMHutkGQEQX5ydbO3A8OIY/vXq+6Si5YsbraERkk4jsEJEdfX19aY5ljsvpwIOfWoNV80rx5cd34fXD/aYjERGAlw/24r/8dDcubyzH9z65Bk4Hd74SERER5apYXPHI5jasqS/FZQ1lpuOQRYkI7n1fE1r7Qvj9oV7TcYiIjIrG4mgPhGwzBODzuFBbmo+jfSHTUSjHTTsEoKp3qmqNqrpVtU5VH1bVB1X1weTjqqpfVNX5qnqpqu5If+zZ86XrFqA4341v/no/VDlVSUQXZjAcwf958TCuXODH9UuqTMfJFTNeR6OqD6nqWlVdW1lZmZFwpvg8LvzornVorizApn/ZgbeODZiORJTTtrcFcd+/7sQlNUV4+LNr4XU7TUciIiIiCxKRm0TkHRE5IiJfO8Pjd4lI3ylVzPeayEkX78UDJ9ERCOPe9zWbjkIWd8ulNagu9uKHr7HFlohy2/GBMUzGFPOTJ+ztoLmygE0AZJz1l2ekWanPg/98wyJsPhLAiwc4VUlEF+YfXjyMkfFJ/M8/WgoRnvDMENuuo0m3Ep8bj92zDpVFefjsI9s5CEBkyPa2IO7+0XbUleXjx3evQ5HXbToSERERWZCIOAH8E4CbASwFcKeILD3DpT89pYr5hxkNSbPm4dfaUFeWjxuXzjEdhSzO7XTgrisb8cbRAPZ1DZmOQ0RkzNHexM30+VX2aAIAgPmVhTjaO8rDx2RUzg8BAMAn19djQVUhvvXr/YhE46bjEJHNHOkdwb9s7cAd6+qxpKbYdJyskc3raDKhqsiLf/v8BpQXePDpH27Dm+1B05GIcsobR/rx2Ue2Y06JFz+5dwP8hXmmIxEREZF1rQNwRFVbVTUC4AkAtxnORGmw5/ggtrcHcfeVTXA5+bIsTe/Oy+vh8zjx8OtsAyCi3JU6UT+/wkZDAFWFCEVi6BkeNx2FchifbSIxVfn/fGgp2gNhPLal3XQcIrKZb/36AHxuJ/7iA4tMR8kq2b6OJhNqS/Px000bMafEi888vB1vHO03HYkoJ7xyqA93P/om5pUnfgarS7ymIxEREZG11QI4fsrHncnPne5jIrJXRJ4UkXlneBwisklEdojIjr6+vnRkpYvww9fbUJTnwh+vrTMdhWyixOfGH6+dh2f2dOEkbyQRUY462jeKisI8lPjs07CYWl1wtDdkOAnlMg4BJF2zqBLvX1yJf3jxMI4Hw6bjEJFNPPt2N15+pw9fun4BT3mSJVWXePHEpg2oK8vH3T96E79p6TEdiSirPbOnC5//8Q40Vxbi8c9vQGUR/20gIiKiaZ1pp9zp3bHPAGhU1RUAXgTw4zN9IVV9SFXXquraysrKWY5JF+N4MIxn3+7Gf7p8HtdE0Xm5+8pGROOKRzazDYCIctPRvtDUTXW7WFCZaC1ItRgQmcAhgFM8cNtyCICv/nQ3ojGuBSCic+saHMPX/n0vVtaV4O4rm0zHITqrqiIvfvqnG7Gkphh/9pOdbL0hSpMfvtaKLz2+C6vmleKJz3MFABEREc1YJ4BTT/bXAeg69QJVDajqRPLDHwC4LEPZaJZ856XDcDoEn7+62XQUspkGfwFuXTkXj73Rgf7Rien/AyKiLKKqONI7ivlV9lkFAACVRXkoynNxCICM4hDAKeaV+/Ctj16KnR0D+O7vjpiOQ0QWFosrvvrT3YjFFf94x2q4ucuPLK68wIPHP78B118yB9/45T78f88dRDx++uEiIroQsbjigWf245u/PoBbLq3GY/ess1VFHRERERn3JoCFItIkIh4AdwB4+tQLRKTmlA9vBXAgg/noIrX1h/CLXSfwqfUNmFPMVVF0/r58/UJMRGP4v68cNR2FiCijgqEIhsYmMb/SXkMAIoLmqkIOAZBRvGt1mltXzsVH19Tiu787jDfbg6bjEJFFff/3R7C9LYi/vm05GivsVUVEuSvf48SDn1qDP1lfjwdfOYr7/nUnRieipmMR2drw+CQ+/9gOPLK5DXdf2Yjv3bkGXrfTdCwiIiKyEVWNAvhzAM8jcXP/Z6q6T0QeEJFbk5d9WUT2icgeAF8GcJeZtHQhvvvSYbidgvuuZQsAXZj5lYW4fXUtHtvSgd7hcdNxiIgy5mhfCABstw4ASGQ+2hsyHYNyGIcAzuCB25ajrsyHrz6xGwFWLBHRad5sD+L/vHgYH145Fx9bU2s6DtF5cTkd+Obty/GNDy3FSwd78dF/3oyOAJ+MEl2Io32juP2fNuPVQ3345u3Lcf+Hl8HhONNKXyIiIqJzU9VnVXWRqs5X1W8lP/cNVX06+f7XVXWZqq5U1fer6kGziWmmjvSO4qndJ/CZjY2oKmILAF24L1+3ENG44vtsAyCiHJI6SW+3JgAgkblneJyHsMgYDgGcQWGeC9+9czX6Rydw72M7MBaJmY5ERBZxtG8Un39sB+aV5eObty+HCG/2kP2ICD53VRMe+9w69I5M4NbvbcbLB3tNxyKylRf29eD2723GUHgSP7l3PT61ocF0JCIiIiKyoO+8dBhetxN/ejVbAOjiNFYU4GNravGTbcfQM8Q2ACLKDUd6R+F1O1Bbmm86ynlbUJUYXGjlSgAyhEMAZ7FyXin+8Y7V2H18EF9+Yhdi3JtMlPP6RiZw14+2wymCH39uHUryue+Z7O3KBRV4+otXYW5pPu5+9E387bMHMBmLm45FZGmRaBx//cw+bPqXnWisKMAv//xKrG/2m45FF0BEbhKRd0TkiIh87QyPN4jISyKyV0R+LyJ1pzxWLyIviMgBEdkvIo2ZzE5ERET2cLBnGM/s7cJnr2iEvzDPdBzKAl+6biHiccV3f3fYdBQioow42jeK5opCWzYvptoLjvRyCIDM4BDAOdy0vBr3f2gpfrv/JP76mX1Q5SAAUa4KR6K458dvom9kAg/fdTka/v/27jw+qvre//jrO1uSyb6RQEhCwr4vghE31KpVqmDVKlprqQtdrr29rb239XfbXrvd6+12u2hr3WqxrXu11KW2WjdkV0R2hBBIIJCQkH2bTL6/P2bACCgBk5xkzvv5eOSRmeRM+HwPmXlnzvl8vydz8F2DSORYCjKDPPWl0/l0SQG/fa2Uq3+7nIqDLU6XJTIg7a5p4VN3L+N3b5Sx8PQRPPHF2QxPDzpdlpwEY4wXuAu4GJgAXGOMmXDEZj8BFltrpwDfA/6n2/cWAz+21o4HTgW0nIqIiIi8j7WW7y7ZREq8n0VnaRUA6R35GUGuLSng4VW72VzZ4HQ5IiJ9qqvLsq68jvFDU5wu5aQUZgYJBry8XV7ndCniUmoCOI6FZxSx6OxiFi/fxR1/26JGABEXamrv5MYH17BhTz13XjODaflpTpck0qvi/V5++MnJ/Oqa6Wzb38TFP3+dJ96sUOaJRFlreXjVbi76xWuUHmjm7utmcPu8icT5vE6XJifvVGC7tbbUWtsBPALMP2KbCcBL0dsvH/p+tFnAZ639B4C1tslaq+4pEREReZ/n1u9jeWkNX79wDOmJAafLkRjytQvGkJrg57+WaNKaiMS2d6uaONgSoqQ4w+lSTorf6+GUwnRWltY6XYq4lJoAeuCbF42LzJB8tZRvPb2BLl0aQMQ16lo6+PR9K1lVVsvPrprG+RNynC5JpM9cOnUYz3/lLMYPTeHrj69j0UNvcqCp3emyRBxV1dDGDQ+u5rY/r2dafho8CqNGAAAgAElEQVR/+7ezuWjSUKfLko8uDyjvdr8i+rXu1gFXRG9/Ekg2xmQCY4A6Y8yfjTFrjTE/jq4sICIiIgJAa0eYHz67ifFDU7i2pNDpciTGpAUDfP3jY1m1s5Zn3ql0uhwRkT6zcmcNAKcVDd7LMJYUZbB1fyO1zR1OlyIupCaAHvB4DD+4bBKfn1PMH1fu5muPva1rJou4QFVjGwvuWcHmygbuvu4ULpt+5LkBkdiTnxHk4UWn8Z9zx/Pq1mou+NmrPKlVAcSFrLU8uno35//sVZbtqOH2SyfwhxtLyEtLcLo06R3HupjgkS90XwfmGGPWAnOAPUAn4APOin5/FlAMLDzqHzBmkTFmjTFmTXV1dS+WLiIiIgPdb17Zzt76Nm6/dALeQXgNYxn4FswqYOKwFP77uc20dHQ6XY6ISJ9YWVrL0NR48jMG77GYkuJIA8OqnVoNQPqfmgB6yBjDbReP598/Ppan397L5363moPq3BGJWRv21HP5r5exu7aF3y2cxQVaAUBcxOsx3Hx2Mc/+65kUZydx6+PruO7+lZQdaHa6NJF+saO6iQX3rOAbT65n3NAUnvvKWSw8owiPDuDGkgogv9v94cDe7htYa/daay+31k4H/jP6tfroY9dGLyXQCTwNzDjyH7DW3mOtnWmtnZmdnd1X4xAREZEBpry2hbtfK2Xe1GGHD/yL9Davx/DdeROprG/jrpe3O12OiEivs9aycmcNJUUZGDN4j8dMGZ5KnM9zeFUDkf6kJoAT9C/njuJHV05h1c5aLr1zKRv21Dtdkoj0sqfWVnDFb5YR7rI8sug0zhiV5XRJIo4YnZPM45+fzQ8um8Q75fVc+PPX+MkLWzXLQGJWU3sndzy/hYt+/hqbKxv43ysm88jNpzEyO8np0qT3rQZGG2OKjDEBYAGwpPsGxpgsY8yh90u3AQ90e2y6MebQmf3zgE39ULOIiIgMcF1dlm88+Q4+j+G2ueOcLkdi3MwRGVw+PY97Xitl414doxaR2LKjupkDTR2DvqEuzudlRkE6K0u1EoD0PzUBnISrZubz2BdmE+6yXPGbZTy2plzLJIvEgLZQmNuXbOSrj65jWn4af/3ymUwZnuZ0WSKO8ngM151WyIu3zmHupFzufHk7H/vpqyxZt1fZJzGjq8vy57cqOO8nr3D3qzuYPy2PF2+dw9WzCjT7P0ZFZ/DfArwAbAYes9ZuNMZ8zxgzL7rZOcBWY8w2IAf4YfSxYSKXAnjJGLOeyKUF7u3nIYiIiMgA9PvlZSzbUcO3L5nA0NTBu3SxDB7fvmQC6cEAX330bdpCYafLERHpNYdmzpcUZThcyUdXUpzB5n0N1LeEnC5FXEZNACfp0AnCGQXp/McT77DooTepbmx3uiwROUnvVNRxya+W8uCyMm44o4g/3FRCVlKc02WJDBg5KfH8fMF0Hv/CbDISA/zrw2u57NfLWFGqpaxkcHtj+wHm3bWUrz22jqGp8Tz1pdP5yaemMiQ53unSpI9Za5+z1o6x1o601h46wf8da+2S6O0nrLWjo9vcZK1t7/bYf1hrp1hrJ1trF1prdZ0wERERl9te1cgdz2/hvHFDWDAr//gPEOkF6YkB/vfKKWzb38RP/77V6XJERHrNytJaspPjKMpKdLqUj6ykKBNrYXWZVgOQ/qUmgI8gKymOP95Uwrc+MZ5Xt1Vz4f+9yl81M1JkUGkLhfnZ37fyyV8vo6mtk8U3nMp3Lp2A36uXR5FjmTUigyW3nMmPrpzC/vo2FtyzghseXK2lB2XQWV9Rz/UPrOLT963kYHOIn101lae+dAbTC9KdLk1EREREBplQuIuvPrqOYMDLHVdMHtTXLpbB59yxQ/h0SQH3Ld2pRn0RiQnWWlbtrKWkKCMmMnV6QRoBr4dVagKQfuZzuoDBzuMx3HRWMeeMzebWx9bx5YfX8vCq3dw+byJjcpKdLk9EPoC1lhc3V/H9Zzaxu7aFy6fn8V/zJpKa4He6NJEBz+sxXDUzn3lTh/G7N8r49Svb+cQvl/LxiTn82/ljGD80xekSRT7Qhj31/PzFbby4uYrUBD//OXc8n5ldSLzf63RpIiIiIjJI/fzFbazfU89vPj1DK0qJI/7f3PEs3X6AWx9bxzNfPpP0xIDTJYmInLTdtS3sa2ijpDjT6VJ6Rbzfy7T8NFaqUUv6maa69pJRQ5J58oun8/35E9m4t4GLf/E6ty/ZyIEmXSJAZKDZXNnAwt+t5ubFawj4PPzhxhJ+dvU0NQCInKB4v5cvnjOSpd84j698bDTLttdw8S9e5+bFa3hr90GnyxN5n9Vltdz44Gou+dVSVu2s5dYLxrD0G+dy89nFagAQERERkZP27DuV3PXyDq6aOZyLJw91uhxxqcQ4Hz+/ehrVje38y5/eIhTucrokEZGTtrI0MmP+tKIMhyvpPSXFGWzY20BTe6fTpYiLaCWAXuTzevjM7BFcMmUYP/n7VhYvL+OxNeUsPH0Ei84uJi2oDkwRJ22vauLnL27jmXcqSY738e1LJnD97EIt/S/yEaUm+PnqBWO44YwiHnhjJw8uK+Mfm/ZTUpTB5+cUc86YIXg8g3/pLhl8wl2Wf26p4rev7mDNroOkB/187YIxLDxjBCnxavwSERERkY9mw556bn38bU4pTOf7l01yuhxxuekF6fz35ZP5+uPr+MEzm/jufP1OisjgtGJnDZmJAUYNSXK6lF5TUpTJr/65nTVltZwzdojT5YhL9KgJwBhzEfALwAvcZ62944jvLwR+DOyJfulOa+19vVjnoJKeGOCHn5zMDWcW8YsX3+U3r+7goeW7uLakgIVnjGBoaoLTJYq4yjsVddzzWinPra8k3u/llnNHcfNZxaQGdQJIpDelBiPNAIvOLuaR1eXc93opNzy4huKsRBaeMYIrZgwnMU79h9L3GttCPL6mgt8vL2NXTQt5aQncfukErpqVTzCg30ERERER+eiqG9tZtHgN6cEAd193CnE+rS4lzrvylOFsqWzgvqU7GZubwrUlBU6XJCJyQqy1rCyt5dSiDIyJnUlFMwrT8HkMy3fUqAlA+s1xj4IaY7zAXcAFQAWw2hizxFq76YhNH7XW3tIHNQ5aI7OT+OU10/nSuSP51T+3c+/rpdy/dCfzpg7j+tNHMHV4aky9iIkMJJ3hLl7aUsUDS3eycmctyXE+bj6rmEVnF5OZFOd0eSIxLTHOx41nFnH97EKeW1/JA2+U8Z2/bORHf9vKJ6fncW1JAeOHpjhdpsSgDXvq+dOq3fxl7R6aO8KcUpjOf3x8HB+fmINPq76IiIiISC9pbAtx0+I11LZ08MQXTic7WccZZOC4be543q1q4jt/2cDQtHjO1ckmERlElm4/wJ66Vm69cIzTpfSqYMDHmaOz+PPaPdx64VgCPh2nkr7Xk6lQpwLbrbWlAMaYR4D5wJFNAPIBxuWmcNe1MyivbeGBN3by6Opy/rx2DxOGRrox500bpiVpRXpJxcEWHltdzqNrytnf0M6w1Hi+9YnxXD0rn2Q9z0T6ld/rYf60POZPy+Ot3Qf5w/JdPLqmnIdW7GJGQRqfmpnPJVOG6rkpH0l9a4i/rtvL429WsK68jjifh0unDuMzpxUyNT/N6fJEREREJMY0toX47AOr2Linnl9/egaT8lKdLknkfbwewy+vmc61967g8w+9yT2fOUWzTkVk0Pj9sjIyEwN8YspQp0vpdZ+dPYLPPbia5zdUMn9antPliAv0pAkgDyjvdr8CKDnGdlcYY84GtgFftdaWH2MbV8vPCPJfl07kaxeM4em39/Knlbv51tMb+P4zmzh/fA6XTc9jzphsdQCJnKD6lhDPrq/k6bV7WFVWizFwzphsfnBZIeeOzdbsT5EBYEZBOjMK0vn2JRN48q0KHlldzm1/Xs93/7qRiybmctn0PM4YlYVfz1fpgVC4i9ffrebptXt5YeM+2ju7GJuTzHcumcAVM4brci8iIiIi0iea2jtZ+LvVrKuo565rp3PhxFynSxI5ptQEP3+8qYRr713Joofe5N7rZzJnTLbTZYmIfKjdNS28tKWKW84dFZOX2ZkzJpsRmUF+v6xMTQDSL3rSBHCs9ertEff/CjxsrW03xnwB+D1w3lE/yJhFwCKAggL3Xo8oOd7PZ04r5LqSAtZV1PPUWxU8804lz66vJDnex/njc/j4xFzmjMkmIRB7L3QiveFAUzt/37ifv23cx7LtB+jssozMTuTrF47hsul5DE8POl2iiBxDemKAm84q5sYzi1hXUc8Tb5az5O29PP32XjITA8ydPJRPTBnKrBEZeD26ZI68pzPcxaqdtTy7vpLn1ldysCVEWtDPgln5XHlKPpPyUnSZJRERERHpM7XNHdy8eA1vl9dx5zXTuWhS7M1QlNiSFgzwx5tK+PR9K7l58RruvEaNKyIysD20ogyvMXy6pNDpUvqEx2O4fvYIvvfMJtZX1DN5uFYTkr7VkyaACiC/2/3hwN7uG1hra7rdvRf432P9IGvtPcA9ADNnzjyykcB1jDFMy09jWn4a37pkAkvfPcCz6yt5cfN+nlq7hzifh9NHZnLuuCGcM2YIBZk6qSnu1dVl2bC3npe3VPPy1irWVdRhLRRmBrnhzCIunTJMJ4BEBpHuGfjtSybw2rYD/OXtPTz+ZuRyAVlJAS6cmMuFE3KYPTIzJrt/5fjaQmGW76jhhY37+Pum/dQ2d5Dg93LBhBzmTxvGWaO1gpKIiIiI9L1Nexu4efEaqpvaufOa6Vw8WQ0AMjikJ0YaARb+bhWLHnqTWy8Ywy3njdLxMxEZcFo6Onl0dTkfn5RLbmq80+X0mStnDucnf9/Kg8vK+OlVU50uR2JcT5oAVgOjjTFFwB5gAXBt9w2MMUOttZXRu/OAzb1apQv4vR7OHTeEc8cNIRTuYmVpLS9u3s/LW6t4+S8bgY3kZyRwxsgsTh+VRUlRBjkpsftCKGKtZXtVEyt21rJs+wGWl9ZQ1xLCGJg6PI1/+9gYLpyYw7jcZL1xERnk4nyRk7oXTMihub2Tl7dW8fyGfTy9dg9/WrmbxICXs8dkR5vishmi/Itp++rbeHVbFS9truL1dw/QGgqTGPBy3vgc5k7KZc7YbIKBnvwJKyIiIiLy0T2/vpKvPbaO1AQ/T3xhNlOGpzldksgJSU8M8OjnZ/PNJ9/hp//YxpZ9jfz4U1P0vkpEBpSn1u6hoa2ThaePcLqUPpUS7+eKGcN5dHU5t80dR1ZSnNMlSQw7btJbazuNMbcALwBe4AFr7UZjzPeANdbaJcC/GmPmAZ1ALbCwD2uOeX6vhzNHZ3Hm6CxuZyI7DzTz6tYq3thRw7PrK3lkdTkABRlBZo3I4JTCdKblpzEmJ0nXPpdBq7UjzPo99azdfZA1uw6ypqyWgy0hAIalxnP++BzOGJXJ2aOzyVQwisSsxDgfl0wZxiVThh2eBf6Pzft5afN+nt+wD4DxQ1M4e0wWZ4zMYtaIDF06Z5Br6ehk1c5alu2o4bVt1WzZ1whAbko8V5ySx/njczitOJN4v/6fRURERKT/1LeG+OGzm3hsTQUzCtK4+7pT1JAsg1a838v/XT2NCcNSuOP5LWyqbOCOyydTUpzpdGkiIlhrWbxsFxOGpjCzMN3pcvrcZ08v5KEVu3hk1W5uOW+00+VIDOtRu5+19jnguSO+9p1ut28Dbuvd0uSQoqxEirKKWHhGEZ3hLjbubWB1WS2ry2p5ZWsVT75VAUCC38ukvBQm5aUyaVgqE/NSKM5K0jK5MuA0tXeydV8DG/c2sGFPPev3NLBtfyPhrshVQgozg3xsfA6njsjg1KIMCjODmu0v4kLxfu/hVXLsZZPYXNnIK9uqeGVrNfe/vpPfvlpKwOthekEaJcWZlBRlML0gTbMZBrjm9k7e2n2QVTtrWVlay9ryg4TCFr/XcEphOt+8eBznjM1mbI5WehERERERZ7ywcR/ffnoDNc0dfGHOSL56wWhdokwGPWMMi84eyaS8VL7x5Dtcfc8KrjutgG9cNI7keL/T5YmIi92/dCdb9zfyk09NdcWxoFFDkjl7TDZ3v1rKRZNyGTUk2emSJEbpKPkg4/N6mJqfxtT8NG46qxhrLbtrW3i7vI61u+tYv6eeR1aV0xoqA8DvNYwaksy43GRG5yQxekgyo4ckMTw9QasGSJ9rC4XZeaCZbfsb2V7VxLb9jWzZ18iumpbD22QkBpg4LIXzx488fH1wzfQXkSMZY5gwLIUJw1L40jmjaG7vZFXZe5cLufOf7/JLCz5PZLsZBelML0hjen46+RkJrngDMRBZa9lV08K6ijre2nWQN3cfZHNlpOnLY2DisFRuOKOI00dlMWtEuho4RERERMRRb+6q5Wf/2MYb22sYPzSF+z87i8nDU50uS6RXnT4yixf+7Wx++vdtPPDGTv62YR9fmDOS604r1ApsItLvXttWzX8/t5mLJ+Vy+fQ8p8vpN/9z+WTm37mUmxe/ydP/cgapCWrGkt5nrLWO/MMzZ860a9asceTfjnXhLktpdRObKhvYXNnIln0NbN3XSGV92+Ft/F5DYWYixVmJjMhKpDAzyIjMRPLTgwxNi8evBgHpodaOMBUHW9hd20JZTQu7aprZeaCZ0upm9ta3cuglxusxFGYGGZ+bwrjcZMbmJjMpL5WhqfE6OdeHjDFvWmtnOl3HQKL8iU0NbSHe3HWQ1TtreWv3QdaV19MaCgOQHvQzeXgak/NSmDA0lYnDUijICOLx6LWnN3V1WXbVtrBxb/3hlV7eqainvjVyaZdgwMvU4WnMKEzj1KJMTilMJylOJ/1jkbLnaMoeEZG+p/w5mvKnZ7q6LCtKa7j7tVJe21ZNVlKAL54ziutnF+r4mMS8deV1/PiFrSzdfoDs5Dg+f3YxV54ynLRgwOnSBgVlz7Epf6Snyg40M+/OpQxLS+DJL55OosuOE60uq+Xae1dw+sgsHlg4C6+OVUoP9TR/3PWMcgmvxzA6J5nROcnMn/be1xvbQmyvauLdqiZKq5sprW6i9EAzr2yrpqOz632Pz02JZ3h6AnlpCQxLS2BoWjxDU+PJSYlnaGoC6UG/Tty6QGe4iwNNHVTWt7K/oY3K+jb21rWyt66NPXWtVBxs5UBT+/sekxzvY0RmIjNHpFOclU9xdiKjc5IoykrU0nki0mdS4v2cO3YI544dAkRev7bsa2RdRR3rK+pZV1HPb18tpTN62ZHEgJfROcmMzUlmTG4yo4YkMTI7kWGpCWoOOI6uLsueula2VzexI7rKy9b9Tby7v5GWjkjjhd9rGD0kmbmTc5kyPI0pw1MZm5OsVYhEREREZMAor23hL2/v4dE15ZTXtpIe9HPbxeP4zOxCrVAlrjE1P40/3FTCitIa/u8f2/jBs5v50QtbuWhiLlfNzKekOEPNMCLSJw40tXPz4jV4PIZ7r5/pugYAgFkjMvjuvEn8v6fW84NnN/GtT0xQI4D0Kvc9q1wsOd7P9IJ0phekv+/rXV2WfQ1t7Kppoby2hfKDkc976lpZubOWfQ1th6/VfkjA6yE7Oe7wR1bSoc8BMhPjyEgMkJkUIC3oJz0Y0B+LA0hbKExtcwcHWzqobe6gpqmDA03t1DR3UN3YzoGmdqob29nf0E5NcztHLhYS5/OQF20MOX/8EPIzggxPTyA/I7KahBpERGQg8Hk9TMpLZVJeKpREvtbeGebd/U1s3FvP5spGtu1v5MXN+3l0Tfnhx8X7PYzITGREZiKFWUEKMoLkpwfJzwgyLC3eNc1MbaEwe+taKT/YGvnboLaFsppmyg60sKu2mbbQe82DWUlxjM1NYsGsAsblJjNhWAqjc5Jcs69EREREeoMx5iLgF4AXuM9ae8cR348DFgOnADXA1dbasv6uczBraAuxrryO1989wMtbqni3qgmA2cWZfP3CsXx8Yq6WQhfXOq04k0c/P5uNe+t5bHU5T63dw5J1e0mO93H26GzmjM3m1BEZFGYGddzPhY6XUSInIhTu4vfLyvjFi+/S3tnFg5+bRX5G0OmyHHNtSQFb9zXwuzfKWFFay3fnTeTUogyny5IYoSYAweMxDIvO+J89MvOo73eGu6hqbGdfQxv76yOzwasa26lqbKOqoZ3y2hbe2nWQ2paOo04YH5Ic5yM12hCQmuAnNcFPSoKPlHg/yfE+UhIin5Pi/CTGeUmOfk6M8xEMeAkGfOqAIhKQLe1hmjs6aW7vpKm9k+b2ME3tIRrbOg9/NLSFaGgN0dAWor41RF1L5PPBlo73nbjpzucxh5s5hiTHMTkvlSEp8QxJjju8CkRuajyZiQH9sS8ig1Kcz/teY0A3NU3tbK9qYkd1Mzuqmyg70My7VY28tGU/ofD7gy0rKcDQ1ARyU+PJSYkjJzmeISmRZrjMpEgzXHowQDDgHXCvldZamjvCHGzuoKa5gwPdG78a29jf0M6+6IovNc0d73tswOuhIDPIiMwgZ43OYuSQpOjqCUlkJGqZSBEREZGPwhjjBe4CLgAqgNXGmCXW2k3dNrsROGitHWWMWQD8L3B1/1c7sFlrqWsJsbe+lbIDLZRWN7Gjuol39tRTWt0MRFatOrUog6tn5XPhhFwKMt174kHkSBOHpfLd+ancNnc8r2yt5uUtVby8tYpn11cCkJrgZ8rwVMblJlOUlURxdiL5GUGyk+II+DQJLBb1MKNEPpS1lm37m1i+4wB/WLmb7VVNzBmTzXcuncDI7CSny3Pc7fMmUlKcyQ+f3cxVv13O3Mm5zJ08lNOKM8lKinO6PBnE1AQgx+Xzeg43CXyYznAXB1tC1DS3U9MUmWVe19JBbXPk5HPkZHQHda0hKutbaWjrpL419L5LEXyYgM8TaQjwe4kPeIn3eYn3e4j3e6MfHgJeDwFf9MPrjX42+LwefF6D3xP57PN68HkMXo/BawweD3hM5L7HGCL9BgZjwMDhEzk22uVgIdrwYAl3QZe1hz/CXZHVFTq7LJ1dXXSGI59DYUtn2BIKdxEKd9ER7qKjs4v2zkOfw7R3dtEWCtMWOvQ5crulo5PWUPiok1EfJBjwRhot4iPNFvkZQSYn+EkL+kkLBshIDJAe9JORGEdmUoDMxAAp8X4tgS0irpQZPYFfUvz+Rrhwl2V/Q1t0lZxW9ta1Ulnfyp66NnbXtLC6rJa6ltAxf2bA6yEt6CclwU9KtNktMc5HUsBHMM5LMJpjCQEvcT4P/mh++bwevMbgjeZS9/w5lDFha+mMZkgoHMmR1o4wbZ1hWjrCNLd30twRpqmtk8a20OG8rW8J0RE+duamB/3kpEQaviblpTAsNZL7+RlB8jMSyEmOV0aIiIiI9J1Tge3W2lIAY8wjwHyg+wmW+cDt0dtPAHcaY4y1HzQd46N5afN+unrpJx9Z4nvHVCL3rH3va0ceXwl3ddHZZQl1vnccJXKcJExrKDIRor41MvGhtrmDqsb2o44zDUuNZ8KwVC6fnsfU/DSmF6ST5MIlh0VORLzfy0WTcrloUi7WWrbub+Tt3XWsq6hjXXk9i5fvov2I51pmYoDs5DhSEvykJUTeDwcDkfe9Qb/vvWO2Pg9+j8HjMYePz5ro8VjzvuOxcOj4bOTWe3qz6X780GSGp6sZ6EP0JKN6RWV9Kxv2NPT2j5Ve0D3Lu58bsTZynCrcFcnuQzndFgpzsCVEVUNkQunmyobDE05GZidy/2dnct64IQNuAo1TjDHMnTyUc8cO4e5Xd3D/0p08t34fAKOGJFGYETw8YTM53kec30t89PXUmOg5LvP+182+es2U3nP++L5/DugvXuk1vm6XCDgR7Z3hwzPYm9sjn5vaO2npiMxyb27vpKUjTEuok9aOyAmOQyfIW0Nh2kNdNLZ10hYK0xHuoj0UOaEeCtvDbxAHEo+JNDT4vZGmhTifhzi/l4DXQ7zfQ5zPS3K8jyHJccT7vST4o38sH/7wkRgX+ZwU5yMp3kdiwEdyvC+6moJP11wWEekF3m4r5ZR8wDZtoTDVje2HZ9bXNLdzsCXS/FbXHKKxPURDaye1zR3srm2JnKBvD9PS0dlrB1W715vg9x5eSScpzkdqgp/c1HiS4/ykRxvA0oORS/YcWrkgKylOy56KiIiIOCsPKO92vwKO+hP08DbW2k5jTD2QCRzovpExZhGwCKCgoOCkC/rSH9866gTfQOExEAz4SAh4SYqLNNymBwMUZyUebmzNSYlnRFaQoqxEggEd/hT5KIwxjMtNYVxuCgtOjbyudHVZ9ta3UlrdzN661sgKsg2R1ebqW0PsqmmhoS0UadjpCH9gQ/pA8D+XT+aaU0/+9dIFepJRvZI/q3bW8pVH3j6px8rAc+iS0kNS4pgzJpvTRmYyuzjT1Uv/H09CwMtXLxjDl88bxfo99SwvreHNsoPsrW9jXUUdB5o6jv9DZNAo/e+59HV/hv4KFsfF+bzEJXn7bFkTG+1E6+yKzMLvDNvoLMrIDP2urve61SIzLSOzP220k+2DmGh3qtcTmbF5qOPK262T9VA366GVB/xejy5rICISQ+L93uhs+RN7A2OtJRS2tHVGmtkOrRITCndFZzxFOqi7O7RijdcDfq/n8Eec30OC34tfDWAiIiIig9WxDhQceUSiJ9tgrb0HuAdg5syZJ912+uQXTz/Zh/bIoWMqh28fPsZCdEbwe8dUvNHjKZHVHj34vUYz2kQc5vEYhqcHezyDPtwVmSHc0dlFezgcOVYbfm/2cJfl8HHZQ8dkbfQ28KHHaD+q461+K/2XP3PGZPPMl888mYdKPzPdVu7ovrryoRnq8X7vgLxU5mDh83qYXpDO9IL09329M9xFS3SCbHsoskqSte+tVm27vYbKwNcfTw81AUjMM8ZELwGAZjqKiMiAYIwh4DORaybGO12NiIiIiDisAsjvdn84sPcDtqkwxviAVKC2rwqalJfaVz9aRFzI6zEkRC8NAHx7CQ0AAAohSURBVH6ny5ET05OM6hVpwQBpwUBf/GiRmODzekjxekiJ1+uo9IymjImIiIiIiIiIiDhnNTDaGFNkjAkAC4AlR2yzBPhs9PaVwD+t1TwvERHpcz3JKBERGYC0EoCIiIiIiIiIiIhDrLWdxphbgBcAL/CAtXajMeZ7wBpr7RLgfuAhY8x2IisALHCuYhERcYsPyiiHyxIRkR5QE4CIiAxYxpiLgF8QeZNxn7X2jiO+XwD8HkiLbvNNa+1z/V6oiIiIiIjIRxB9H/PcEV/7TrfbbcCn+rsuERGRY2WUiIgMfLocgIiIDEjGGC9wF3AxMAG4xhgz4YjNvgU8Zq2dTmQmzK/7t0oREREREREREREREZGBRU0AIiIyUJ0KbLfWllprO4BHgPlHbGOBlOjtVGBvP9YnIiIiIiIiIiIiIiIy4OhyACIiMlDlAeXd7lcAJUdsczvwd2PMl4FE4Pz+KU1ERERERERERERERGRg0koAIiIyUJljfM0ecf8a4EFr7XBgLvCQMeaobDPGLDLGrDHGrKmuru6DUkVERERERERERERERAYGNQGIiMhAVQHkd7s/nKOX+78ReAzAWrsciAeyjvxB1tp7rLUzrbUzs7Oz+6hcERERERERERERERER56kJQEREBqrVwGhjTJExJgAsAJYcsc1u4GMAxpjxRJoANNVfRERERERERERERERcS00AIiIyIFlrO4FbgBeAzcBj1tqNxpjvGWPmRTe7FbjZGLMOeBhYaK098pIBIiIiIiIiIiIiIiIirmGcOldijKkGdp3gw7KAA31QzmCh8bt7/KB9oPGf+PgLrbVa/76bk8gft//egfaBxq/xu3n8cOL7QNlzBL33OSkav7vHD9oHGr/e+3xkyp+TovFr/G4eP2gf6L1PL9CxtxPm9vGD9oHGr/H3yXsfx5oAToYxZo21dqbTdThF43f3+EH7QON39/idov2ufaDxa/xuHj9oHzjF7ftd43f3+EH7QON39/id5PZ9r/Fr/G4eP2gfuH38TnH7fnf7+EH7QOPX+Ptq/LocgIiIiIiIiIiIiIiIiIiISIxQE4CIiIiIiIiIiIiIiIiIiEiMGGxNAPc4XYDDNH5x+z7Q+MUJ2u/aBxq/u7l9/KB94BS373eNX9y+DzR+cYrb973G725uHz9oH7h9/E5x+353+/hB+0Djd7c+G7+x1vbVzxYREREREREREREREREREZF+NNhWAhAREREREREREREREREREZEPMOCaAIwxFxljthpjthtjvnmM78cZYx6Nfn+lMWZE/1fZt3qwD75mjNlkjHnHGPOSMabQiTr7yvHG3227K40x1hgzsz/r62s9Gb8x5qro78BGY8yf+rvGvtaD50CBMeZlY8za6PNgrhN19gVjzAPGmCpjzIYP+L4xxvwyum/eMcbM6O8aY5Xb88ft2QPKH7fnj5uzB5Q/TlL+uDt/lD3uzh5wd/4oe5zj9uwB5Y/yx9354+bsAeWPk9yeP8oed2cPKH+UPw7kj7V2wHwAXmAHUAwEgHXAhCO2+RJwd/T2AuBRp+t2YB+cCwSjt78YS/ugJ+OPbpcMvAasAGY6XXc///+PBtYC6dH7Q5yu24F9cA/wxejtCUCZ03X34vjPBmYAGz7g+3OB5wEDnAasdLrmWPhwe/64PXt6ug+i2yl/YjB/3J490TEpf5zZ78ofF+ePssfd2XMC+yBm80fZ49h+d3X2nMA+UP4of2Iyf9yePdExKX+c2e+uzh9lj7uz5wR+B5Q/yp9ezZ+BthLAqcB2a22ptbYDeASYf8Q284HfR28/AXzMGGP6sca+dtx9YK192VrbEr27AhjezzX2pZ78DgB8H/gR0NafxfWDnoz/ZuAua+1BAGttVT/X2Nd6sg8skBK9nQrs7cf6+pS19jWg9kM2mQ8sthErgDRjzND+qS6muT1/3J49oPxxe/64OntA+eMg5Y+780fZ4+7sAZfnj7LHMW7PHlD+KH/cnT+uzh5Q/jjI7fmj7HF39oDyR/njQP4MtCaAPKC82/2K6NeOuY21thOoBzL7pbr+0ZN90N2NRDpDYsVxx2+MmQ7kW2uf6c/C+klP/v/HAGOMMW8YY1YYYy7qt+r6R0/2we3AdcaYCuA54Mv9U9qAcKKvEdIzbs8ft2cPKH/cnj/KnuNT/vQN5Y+780fZ4+7sAeXP8Sh7+obbsweUP8ofd+ePsuf4lD99w+35o+xxd/aA8kf5c3y9nj++j1RO7ztWV5c9iW0Gsx6PzxhzHTATmNOnFfWvDx2/McYD/B+wsL8K6mc9+f/3EVkW5hwi3YCvG2MmWWvr+ri2/tKTfXAN8KC19qfGmNnAQ9F90NX35Tku1l8DneL2/HF79oDyx+35o+w5vlh+DXSS8udobsofZc/R3JQ9oPw5nlh+/XOS27MHlD/Kn6O5KX+UPccX66+BTnF7/ih7juam7AHlj/Ln+Hr9NXCgrQRQAeR3uz+co5d7OLyNMcZHZEmID1s+YbDpyT7AGHM+8J/APGttez/V1h+ON/5kYBLwijGmjMh1MZYYY2b2W4V9q6fPgb9Ya0PW2p3AViLBECt6sg9uBB4DsNYuB+KBrH6pznk9eo2QE+b2/HF79oDyx+35o+w5PuVP31D+uDt/lD3uzh5Q/hyPsqdvuD17QPmj/HF3/ih7jk/50zfcnj/KHndnDyh/lD/H1+v5M9CaAFYDo40xRcaYALAAWHLENkuAz0ZvXwn801obK91g0IN9EF0W5bdEgiCWrgkCxxm/tbbeWptlrR1hrR1B5No486y1a5wpt9f15DnwNHAugDEmi8gSMaX9WmXf6sk+2A18DMAYM55IGFT3a5XOWQJcbyJOA+qttZVOFxUD3J4/bs8eUP64PX+UPcen/Okbyh9354+yx93ZA8qf41H29A23Zw8of5Q/7s4fZc/xKX/6htvzR9nj7uwB5Y/y5/h6PX8G1OUArLWdxphbgBcAL/CAtXajMeZ7wBpr7RLgfiJLQGwn0gW2wLmKe18P98GPgSTgcWMMwG5r7TzHiu5FPRx/zOrh+F8ALjTGbALCwL9ba2ucq7p39XAf3Arca4z5KpHlUBbGyh+ExpiHiSz3k2Ui1775L8APYK29m8i1cOYC24EW4HPOVBpb3J4/bs8eUP64PX/cnj2g/HGK8sfd+aPscXf2gPJH2eMMt2cPKH+UP+7OH7dnDyh/nOL2/FH2uDt7QPmj/HEmf0wM7T8RERERERERERERERERERFXG2iXAxAREREREREREREREREREZGTpCYAERERERERERERERERERGRGKEmABERERERERERERERERERkRihJgAREREREREREREREREREZEYoSYAERERERERERERERERERGRGKEmABERERERERERERERERERkRihJgAREREREREREREREREREZEYoSYAERERERERERERERERERGRGPH/AcPrZLv2WWFQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 2592x360 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from scipy.stats import beta\n",
    "x = np.linspace(0.01,0.99, 100)\n",
    "fig, axes = plt.subplots(1,5,figsize=(36,5))\n",
    "fig.suptitle(r\"$\\beta(\\alpha,\\alpha)$ Distribution\",fontsize=16)\n",
    "alphas = [0.2,0.75,1,10,100]\n",
    "for a, ax in zip(alphas,axes.flatten()):\n",
    "    ax.set_title(r\"$\\alpha={}$\".format(a))\n",
    "    ax.plot(x, beta.pdf(x, a, a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mixup(a_x,a_y,b_x,b_y,alpha=0.75):\n",
    "    l = np.random.beta(alpha,alpha)\n",
    "    l = max(l,1-l)\n",
    "    x = l * a_x + (1-l) * b_x\n",
    "    y = l* a_y + (1-l) * b_y\n",
    "    return x,y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sharpening\n",
    "\n",
    "$$Sharpen(p,T)_i := \\dfrac{p_{i}^{1/T}}{\\sum_{j=1}^{L}{p_{j}^{1/T}}}$$\n",
    "\n",
    "The authors sharpen the model's predictions on the unlabeled data with the above equation as a form of entropy minimization. If the temperature $T < 1$, the effect is to make the predictions more certain, and as $T$ drops towards zero the predictions approach a one-hot distribution (see figure below). This relatively simple step, which involves no learned parameters, turns out to be incredibly important to the algorithm. In an ablation study, the paper reports an accuracy reduction of over 16% when removing the sharpening step (setting $T$ to $1$). \n",
    "\n",
    "The idea behind entropy minimization in semi-supervised learning is that the decision boundary of the classifier should not pass through high density regions of the data space. If this were the case, the boundary would split data that are very close together. In addition, small perturbations would result in large changes in predictions. As predictions near the decision boundary are more uncertain, entropy minimization seeks to make the model more confident in its predictions thus moving the boundary away from the data. While other approaches [3] add an entropy term to the loss, MixMatch directly lowers the entropy of the unlabeled targets via the equation above.  \n",
    "\n",
    "As an example of this technique, let's try a classification problem that's simpler and easier to visualize than CIFAR -- MNIST. We'll still take 500 random examples as the labeled training set, and reserve the rest as the unlabeled set. The full images are used for training, but we'll also reduce each image to two dimensions using tSNE for visualization. Training in a semi-supervised manner following the same approach as MixMatch with regards to the unlabeled data, we'll use the model itself to generate pseudo-labels. The model consists of just two convolution layers and a linear head. No mixup or data augmentation is used, so we can isolate the effects of entropy minimization. The loss function is also largely the same as MixMatch, using cross-entropy for the labeled data and mean squared error for the unlabeled data, see the loss section below for the rationale behind this. The upper image is trained without using sharpening and in the lower image the pseduo-labels were sharpened with $T=0.5$. Training each for ten epochs, the unsharpened model has a test accuracy of 80.1%, and the sharpened model has an accuracy of 90.7%. In the images below, colors correspond to predicted class, and marker size is inversely proportional to prediction confidence (smaller markers are more confident). As shown by the marker sizes, the unsharpened model has a lot of uncertainty, especially around the edges of the clusters, while the sharpened model is much more confident in its predictions.\n",
    "\n",
    "![No sharpening](images/sharpening_none.png \"No sharpening\")\n",
    "\n",
    "![Sharpening](images/sharpening.png \"Sharpening\")*The effect of sharpening on the semi-supervised training of MNIST. Images in MNIST were reduced to two dimensions using tSNE. Colors correspond to predicted class, and marker size is inversely proportional to prediction confidence (smaller markers are more confident). The upper image was trained $T=1$, and the lower image with $T=0.5$.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sharpen(p,T=0.5):\n",
    "    u = p ** (1/T)\n",
    "    return u / u.sum(dim=1,keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABWoAAAFTCAYAAACgdDUEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X28ZXddH/rPlxkCyoOAGXslyTAjDl6DD6QOwdaqVIMkWhPaIiaWCm0gF2tERb0ExYDRVoRb7INpIUKuiA2RB2tHO9wUBVTU0BkkYDM0MmQGMoaWIYk8kzDhe//Ya3Tn5Jw5e2b2OevMOe/363Ves9dav7XWd6+991mzP+e3fqu6OwAAAAAAjOcBYxcAAAAAALDRCWoBAAAAAEYmqAUAAAAAGJmgFgAAAABgZIJaAAAAAICRCWoBAAAAAEYmqAWADaKqXlpVv7FK+/qhqvrfVfXpqvryqvqWqvrgMP201ahhibpOqI7VPHYnq6reWlXPGruOMVXVtqrqqtq8Att+Z1U9Z97bXU+O9/MyvFZfPTx+VVX97Jzq2Dp81jcN03N97XzWAIB5m/t/XgGAcVTVp6cmvzTJ3UnuHab/r1Ws44FJXpnkm7v7fcO8q5L8Snf/25Pc9sEkz+nu3zvBTcyljrWsuy8YuwZObXP4nJ2w7n7eLO1mqbG7P5LkofOoq6pemuSru/uZU9v3WQMA5kqPWgBYJ7r7oUd/knwkyfdOzftPq1jK30ry4CQ3T817zILpsYxex0r08jzVOSarZ6Mc643yPAGA9UVQCwAby2lV9etV9amqurmqdh5dUFWPrqq3VNXhqjpQVc9faiNV9aCq+n+q6iPDEAevqqovqarHJbllaPZXVfX2qvpQkq9K8jvDZcgPqqovq6rXVtVHq+ovq+oXjl6ePGz/uVX1gaHOfVX1t6vq9Um2Tm3n/16itudW1f6qurOqdlXVo4f596tjkXVfONTzqaq6paq+c8Zjd0VVfWiq3n84tezZVfXHVfXLVXVnkpdOzfv3VfWJqvqf0/s61vEZ1n3XcPzvGl6rC6bW/evLu2dou72q/nCo+/eq6upjXbK+1LEdlnVVPa8mQ0vcNWyrltjOS6vqzVX1G1X1ySTPrqpzq+pPq+qvhuf9K1V12izbr6pNw3P8eFXdmuR7Fuzv0UO9dw71P3dBLW8aavlUVf15VT2uql5UVR+rqtuq6rsWeQ4PGrb39VPzvqKqPldVWxZpv6mq/vVQ44Gqurymhmc4ydd8uXUXvv8eW5PP5h1DPf+pqh4xtF/0c1ZV31xVfzK8Pu+rqidP7X97Vf3BcPzeluT0xV73qfY/NdR6e1X98wXLfq2qfmF4fHpV/e6wzzur6o+q6gGL1Vh/M9zFpVX1kSRvr8WHwHhsVf33mnzu/ktVPWrY15Or6tCCWg5W1XlVdX6Sn07y/cP+jl4pMP1Ze0BVvbiqPjy8b369qr5sWHa0jmfV5Hfmx6vqZ451jACAjUlQCwAby4VJrk/yiCS7kvxKMgkZkvxOkvclOSPJdyb5sap66hLb+aUkj0vyhCRfPaxzZXf/RZLHD20e0d3f0d2PzX17+N6d5HVJjgzrnpPku5IcDTy+L8lLk/xgkocPNd/R3f90wXZevrCoqvqOJL+Y5BlJvjLJh4fnmyXqmF73a5JcnuSJ3f2wJE9NcnC5Yzf4UJJvTfJlSX4uyW9U1VdOLX9SkluTfEWSf7lg3ulJXpLkt46GRsc6PlPr3jKs+/Ikr61aPBRdpu11Sf57ki/P5Jj/0yW2ccxjO+UfJHlikm8c2i31/kmSi5K8OZPj+Z8yGabjx4c6/04m78F/MeP2nzssOyfJziRPX7DeG5IcSvLoYdm/qvuG8N+b5PVJHpnkvUluyOT/yWdkMlzGqxcWP7x/rk/yzKnZlyT5ve4+vMjzfW6SCzL5zPztJAvHSD6Z13yWdafff5XJa/noJF+b5KxMXv8s9jmrqjOS/Nckv5DkUUl+MslbpgLp65K8Z6jt55MsOW7rEHr+ZJKnJNmR5Lyl2ib5iUxety2Z9NT/6UmJx/xd8O3Dc1rqvfeDSf758NyPJPl3x9h/Mtnh/5fkXyX5zWF/37hIs2cPP38/kz8IPTT3/R2RJH8vyddk8t6+sqq+drl9AwAbi6AWADaWd3X37u6+N5Ng6mjg8MQkW7r7qu6+p7tvTfKrSS5euIEhHHpukh/v7ju7+1OZhBj3a7uYqvpbmQRWP9bdn+nujyX55an1n5Pk5d29pyf2d/eHZ3x+/yTJtd39Z0OQ9qIkf6eqts2w7r1JHpTk7Kp6YHcf7O4PTS1f6tilu9/U3bd39xe7+zeTfDDJuVPr3t7d/767j3T354Z5H0vyb7r7C8M6tyT5nhmOT5J8uLt/dajldZkEp39riee1aNuq2prJ637l8Jq/K5MAeimzHNuXdfdfDWODviOTUHIpf9rdvz0cs89193u6+8bhGB3MJBz99gXrLLX9Z2RyLG/r7jszCSGTJFV1ViYB2Qu7+/PdfVOS1+S+ofQfdfcN3X0kyZsyCQZf1t1fyCSM3Xa0x+kCr0vyA8MfOjJs8/VLPN9nJPm33X2ou+9K8rKpGk/4NZ9x3fu8/4bP1Nu6++4hVH5l7n+spz0zye7h/f/F7n5bkr1JvnvqffSzw/b+MJM/+izlGUn+3+7+H939mQwB8RK+MDzPxwyfkz/q7j5G+yR56XAcPrfE8tdP7ftnkzyjpnrzn4R/kuSV3X1rd386k8/HxQt68/7ccPzfl8kfxRYLfAGADczYTQCwsfyvqcefTfLgIUh4TJJHV9VfTS3flOSPFtnGlkxuVvaeqU6cNbSfxWOSPDDJR6fWf0CS24bHZ2XSQ/VEPDrJnx2d6O5PV9UdmfSMPHisFbt7f1X9WCbB0eOr6oYkL+ju24cmix677j5SVT+Y5AVJtg3LH5r7Xv59W+7vLxeETh8e6l/u+Nynlu7+7NBuqZsmLdX29CR3dvdnF9R51hLbmeXYLjxGx7qR032OSU2GzXhlJj1ivzST/6e+Z6nnsmD7j16wvelg/9GZPM9PLVi+c2r6f089/lySjw+B6NHpDPua/nyku99dVZ9J8u1V9dFMerQuFXYvrHH68cm85o+aYd2Fx/orMulJ+q1JHja0v2uJuo/W931V9b1T8x6YSVj+6CR3DcHnUR/Osd9H71nQdimvyOTz+N+G53ZNd7/sGO2TxT9rSy3/cCbP45hDNczo0bnvc/lwJu/h6T+gHM/nAwDYgAS1AEAyCS8OdPeOGdp+PJPw6vHd/ZcnuK+7k5w+9GBcbPljl1h3ud50t2cSKiVJquohmVzWP1Od3X1dkuuq6uGZ9Oj8pRxjOIBhH4/JpPfxd2bSS/Teqropk/D6WHWfUVU1FdZuzSTkW+74zMtHkzyqqr50KqxdKlxLTvLYLmLhMfmPmQw7cEl3f2oIzRcOYbCUj+a+tW+denx7Js/zYVNh7daceN0LvS6THqf/K8mbu/vzx6jxzKnp6XpP5jWfZd2Fx/oXh3nf0N13VNXTct/L9Be2vy2TnqjPXTD/6Pv/kVX1kKmwdusi2zjqWK/VfYuevF4/keQnqurxSd5RVXu6+/ePsf3lfkcs3PcXMvmd9plM/kCQZDKmcCZ/lJp1u/f5fAzbPpLJHwHOXHQNAIAFDH0AACSTcUo/WZObaX1JTW589HVV9cSFDbv7i5kEk7889MxLVZ1xjPFsF67/0ST/Lcm/rqqHDzfheWxVHb30+jVJfrKqvqkmvnoIg5JJ6PFVx9j8dUn+WVU9oSY3C/tXSd49XEp/TFX1NVX1HcN6n88kjL53mdWS5CGZhDiHh+38syRfN8N6X5Hk+VX1wGFc3q/N5PLy5Y7PXAzDSezN5OZSp1XV38lkrNalnPCxndHDknwyyaer6v9M8kPHse4bMzmWZ1bVI5NccXRBd9+W5E+S/GJVPbiqviHJpZmMizsPr0/yDzMJa399mRp/dPisPCLJC6dqPOHX/ATXfViST2dyw78zkvzUguULP2e/keR7q+qpw++GB9fk5ltnTr2Pfm54H/29HPt99MZMbh53dlV9aSbjMy+qqv7B8PmvTN4b9+ZvPpPL/S5YyjOn9n1VJuH6vUn+IpNe8t9TVQ9M8uJMhkI56n9nMgTGUt+f3pDkx2tyY7WH5m/GtF3JP7YAAOuMoBYAyBBUfG8mY34eyKSH2WsyuTnWYl6YZH+SG6vqk0l+L5Ob5MzqB5OclmRfJpdcvzmTsSjT3W/K5IZH1yX5VJLfzuTy7mTSE/DFNbkL/E8u8jx+P5NxJ9+SSc+9x2bGsXMzCWVelslz/1+ZBKk/vdxK3b0vyb9O8qeZhDlfn+SPZ9jfuzO5mdLHM3m+T+/uO4ZlSx6fOfsnmdy4645MbhT1m5n0zryfkzy2s/jJJD+QyWv+q0Mts/rVTG4A9r5Mhmf4rQXLL8lkWIrbk/znJC8Zxlk9ad19aNhnZ/GhQqZr/G9J3p9Jz+HdmfS4PBo8nsxrfrzr/lwmNzT7RCY3CVt4vO7zORvC7osy+TwczqSH7U/lb75L/EAmNyy7M5PgdcnAurvfmuTfJHl7Jr9D3n6MOndk8rvl05l8vv5Dd79zsRqPsY2FXp/k1zL5jD84yfOHuj6Ryc3rXpNJb+vPZHIjs6PeNPx7R1X9We7v2mHbf5jJ79DPJ/mR46gLACC1/Hj8AADMU1U9O8lzuvvvjV3LtKr6zST/s7uX7OXI/VXVtZncsOvFx7HOBUle1d2PWbYxAAAbgh61AAAbVFU9cbhM/gFVdX4mvSZ/e+y6TiVVtS3JP0ry2mXafUlVfXdVbR6GG3hJJr17AQAgiaAWAGAj+z+SvDOTS8v/XZIf6u73jlrRKaSqfj7J/0jyiu4+sFzzTIYcuCuToQ8+kOTKla0QAIBTiaEPAAAAAABGpkctAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQCwAAAAAwMkEtAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQC3NWVc+qqk8PP5+vqnunpv+qqh50Etu+vKr2VtXdVfVry7R9VFX956r6TFV9uKp+4ET3CwBrwQqfY2c+b1bVO4f9H933LSe6XwBYC9bQOXbm77ywHlV3j10DrFtVdWWSb+rui+a0vX+U5ItJnprkS7r72cdo+4ZM/hhzaZInJPmvSf5ud988j1oAYEwrcI6d+bxZVe9M8hvd/Zp57BsA1pKRz7Ezf+eF9UiPWlhZT0jyvnltrLt/q7t/O8kdx2pXVQ9J8o+T/Gx3f7q735VkV5J/Oq9aAGBkczvHOm8CwH2Mdo6d9TsvrFeCWlhZT0hy02ILqup3h0tIFvv53ZPc7+OS3NvdfzE1731JHn+S2wWAtWKe59gTOW/+YlV9vKr+uKqefMLPAgDWnrHPsbBhbR67AFivqurhSbZliRNcd/+DFdz9Q5N8YsG8TyR52AruEwBWxQqcY4/3vPnCJPuS3JPk4iS/U1VP6O4PHed+AWBNWQPnWNjQ9KiFlfONST6V5MAI+/50kocvmPfwTOoBgFPdvM+xx3Xe7O53d/enuvvu7n5dkj9O8t1zqgUAxjTqORY2OkEtrJwnJHl/L3HHvqp669RdNBf+vPUk9/0XSTZX1Y6ped+YxI3EAFgP5n2OPdnzZiep43oGALA2rbVzLGwohj6AlbPkuD5J0t0XHO8Gq2pzJp/bTUk2VdWDkxzp7iMLtv2ZqvqtJFdV1XOGWi5K8nePd58AsAbN9Rx7POfNqnpEkicl+YMkR5J8f5JvS/Jjx7NPAFijRjvHJrN/54X1So9aWDnfmGOc4E7Qi5N8LskVSZ45PH5x8td/2fzpqbb/IsmXJPlYkjck+aHu9ldLANaDlTjHLnneXHCOfWCSX0hyOMnHk/xIkqd19y1zrgcAxjDmOTY5xnde2Ahqid7sAAAAAACsEj1qAQAAAABGJqgFAAAAABiZoBYAAAAAYGSCWgAAAACAkQlqAQAAAABGtnnsAhY6/fTTe9u2bWOXAcAa9Z73vOfj3b1l7DpORc6xAByLc+yJc44F4FhmPceuuaB227Zt2bt379hlALBGVdWHx67hVOUcC8CxOMeeOOdYAI5l1nOsoQ8AAAAAAEYmqAUAAAAAGJmgFgDWoKo6v6puqar9VXXFIst/uapuGn7+oqr+aow6AQAAmI81N0YtAGx0VbUpydVJnpLkUJI9VbWru/cdbdPdPz7V/keSnLPqhQIAADA3etQCwNpzbpL93X1rd9+T5PokFx2j/SVJ3rAqlQEAALAiBLUAsPackeS2qelDw7z7qarHJNme5O2rUBcAAAArRFALAGtPLTKvl2h7cZI3d/e9i26o6rKq2ltVew8fPjy3AgEAAJgvQS0ArD2Hkpw1NX1mktuXaHtxjjHsQXdf0907u3vnli1b5lgiAJx6quraqvpYVf2PJZZXVf274Wae76+qv73aNQKwcQlqAWDt2ZNkR1Vtr6rTMgljdy1sVFVfk+SRSf50lesDgFPVryU5/xjLL0iyY/i5LMl/XIWaACCJoBYA1pzuPpLk8iQ3JPlAkjd2981VdVVVXTjV9JIk13f3UsMiAABTuvsPk9x5jCYXJfn1nrgxySOq6itXpzoANrrNYxcAANxfd+9OsnvBvCsXTL90NWsCgA1gqRt6fnSccgDYSAS1sIjt2w+Otu8DB7aNtm8AWGkHX7t9tH1vu/TAaPsGThkz39Czqi7LZHiEbN26dSVrgmUd3D7e+TVJth1wjoV5MPQBAAAATMx8Q0837ARg3gS1AAAAMLEryQ/WxDcn+UR3G/YAgFVh6AMAAAA2hKp6Q5InJzm9qg4leUmSByZJd78qk/HhvzvJ/iSfTfLPxqkUgI1IUAsAAMCG0N2XLLO8k/zwKpUDAPdh6AMAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARjZTUFtV51fVLVW1v6quWGT586rqz6vqpqp6V1WdPczfVlWfG+bfVFWvmvcTAAAAAAA41W1erkFVbUpydZKnJDmUZE9V7erufVPNruvuVw3tL0zyyiTnD8s+1N1PmG/ZAAAAAADrxyw9as9Nsr+7b+3ue5Jcn+Si6Qbd/cmpyYck6fmVCAAAAACwvs0S1J6R5Lap6UPDvPuoqh+uqg8leXmS508t2l5V762qP6iqbz2pagEAAAAA1qFZgtpaZN79esx299Xd/dgkL0zy4mH2R5Ns7e5zkrwgyXVV9fD77aDqsqraW1V7Dx8+PHv1AAAAAADrwCxB7aEkZ01Nn5nk9mO0vz7J05Kku+/u7juGx+9J8qEkj1u4Qndf0907u3vnli1bZq0dAAAAAGBdmCWo3ZNkR1Vtr6rTklycZNd0g6raMTX5PUk+OMzfMtyMLFX1VUl2JLl1HoUDAAAAAKwXm5dr0N1HquryJDck2ZTk2u6+uaquSrK3u3clubyqzkvyhSR3JXnWsPq3Jbmqqo4kuTfJ87r7zpV4IgAAAAAAp6plg9ok6e7dSXYvmHfl1OMfXWK9tyR5y8kUCAAAAACw3s0y9AEAAAAAACtIUAsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAa1BVnV9Vt1TV/qq6Yok2z6iqfVV1c1Vdt9o1AgAAMD+bxy4AALivqtqU5OokT0lyKMmeqtrV3fum2uxI8qIk39Ldd1XVV4xTLQAAAPOgRy0ArD3nJtnf3bd29z1Jrk9y0YI2z01ydXfflSTd/bFVrhEAAIA5EtQCwNpzRpLbpqYPDfOmPS7J46rqj6vqxqo6f7ENVdVlVbW3qvYePnx4hcoFAADgZAlqAWDtqUXm9YLpzUl2JHlykkuSvKaqHnG/lbqv6e6d3b1zy5Ytcy8UAACA+RDUAsDacyjJWVPTZya5fZE2/6W7v9DdB5LckklwCwAAwClIUAsAa8+eJDuqantVnZbk4iS7FrT57SR/P0mq6vRMhkK4dVWrBAAAYG4EtQCwxnT3kSSXJ7khyQeSvLG7b66qq6rqwqHZDUnuqKp9Sd6R5Ke6+45xKgYAAOBkbR67AADg/rp7d5LdC+ZdOfW4k7xg+AEAAOAUp0ctAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQCwAAAAAwMkEtAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQCwAAAAAwspmC2qo6v6puqar9VXXFIsufV1V/XlU3VdW7qursqWUvGta7paqeOs/iAQAAAADWg2WD2qralOTqJBckOTvJJdNB7OC67v767n5CkpcneeWw7tlJLk7y+CTnJ/kPw/YAAAAAABjM0qP23CT7u/vW7r4nyfVJLppu0N2fnJp8SJIeHl+U5Pruvru7DyTZP2wPAAAAVt0MV4xurap3VNV7q+r9VfXdY9QJwMYzS1B7RpLbpqYPDfPuo6p+uKo+lEmP2ucfz7oAAACw0ma8YvTFSd7Y3edkcoXof1jdKgHYqGYJamuReX2/Gd1Xd/djk7wwkxPbzOtW1WVVtbeq9h4+fHiGkgAAAOC4LXvFaCbfWR8+PP6yJLevYn0AbGCzBLWHkpw1NX1mjn2iuj7J045n3e6+prt3dvfOLVu2zFASAAAAHLdZrvp8aZJnVtWhJLuT/MjqlAbARjdLULsnyY6q2l5Vp2Vy6ceu6QZVtWNq8nuSfHB4vCvJxVX1oKranmRHkv9+8mUDAADAcZvlqs9Lkvxad5+Z5LuTvL6q7vfd2ZWhAMzb5uUadPeRqro8yQ1JNiW5trtvrqqrkuzt7l1JLq+q85J8IcldSZ41rHtzVb0xyb4kR5L8cHffu0LPBQAAAI5llqs+L01yfpJ0959W1YOTnJ7kY9ONuvuaJNckyc6dO+83xB8AHK9lg9ok6e7dmVzyMT3vyqnHP3qMdf9lkn95ogUCAADAnPz1FaNJ/jKTK0Z/YEGbjyT5ziS/VlVfm+TBSXSZBWDFzTL0AQAAAJzyuvtIkqNXjH4gyRuPXjFaVRcOzX4iyXOr6n1J3pDk2d2txywAK26mHrUAAACwHsxwxei+JN+y2nUBgB61AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCwBpUVedX1S1Vtb+qrlhk+bOr6nBV3TT8PGeMOgEAAJiPzWMXAADcV1VtSnJ1kqckOZRkT1Xt6u59C5r+ZndfvuoFAgAAMHd61ALA2nNukv3dfWt335Pk+iQXjVwTAAAAK0hQCwBrzxlJbpuaPjTMW+gfV9X7q+rNVXXW6pQGAADAShDUAsDaU4vM6wXTv5NkW3d/Q5LfS/K6RTdUdVlV7a2qvYcPH55zmQAAAMyLoBYA1p5DSaZ7yJ6Z5PbpBt19R3ffPUz+apJvWmxD3X1Nd+/s7p1btmxZkWIBAAA4eYJaAFh79iTZUVXbq+q0JBcn2TXdoKq+cmrywiQfWMX6AAAAmLPNYxcAANxXdx+pqsuT3JBkU5Jru/vmqroqyd7u3pXk+VV1YZIjSe5M8uzRCgYAAOCkCWoBYA3q7t1Jdi+Yd+XU4xcledFq1wUAAMDKMPQBAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQCwAAAAAwMkEtAAAAAMDIBLUAAAAAACObKaitqvOr6paq2l9VVyyy/AVVta+q3l9Vv19Vj5ladm9V3TT87Jpn8QAAAAAA68Hm5RpU1aYkVyd5SpJDSfZU1a7u3jfV7L1Jdnb3Z6vqh5K8PMn3D8s+191PmHPdAAAAAADrxiw9as9Nsr+7b+3ue5Jcn+Si6Qbd/Y7u/uwweWOSM+dbJgAAAADA+jVLUHtGktumpg8N85ZyaZK3Tk0/uKr2VtWNVfW0E6gRAAAAAGBdW3bogyS1yLxetGHVM5PsTPLtU7O3dvftVfVVSd5eVX/e3R9asN5lSS5Lkq1bt85UOAAAAADAejFLj9pDSc6amj4zye0LG1XVeUl+JsmF3X330fndffvw761J3pnknIXrdvc13b2zu3du2bLluJ4AAAAAAMCpbpagdk+SHVW1vapOS3Jxkl3TDarqnCSvziSk/djU/EdW1YOGx6cn+ZYk0zchAwAAAADY8JYd+qC7j1TV5UluSLIpybXdfXNVXZVkb3fvSvKKJA9N8qaqSpKPdPeFSb42yaur6ouZhMIv625BLQAAAADAlFnGqE13706ye8G8K6cen7fEen+S5OtPpkAAAAAAgPVulqEPAAAAAABYQYJaAAAAAICRCWoBAAAAAEYmqAUAAAAAGJmgFgAAAABgZIJaAAAAAICRCWoBAAAAAEYmqAUAAAAAGJmgFgAAAABgZIJaAAAAAICRCWoBAADYMKrq/Kq6par2V9UVS7R5RlXtq6qbq+q61a4RgI1p89gFAAAAwGqoqk1Jrk7ylCSHkuypql3dvW+qzY4kL0ryLd19V1V9xTjVArDR6FELAADARnFukv3dfWt335Pk+iQXLWjz3CRXd/ddSdLdH1vlGgHYoAS1AAAAbBRnJLltavrQMG/a45I8rqr+uKpurKrzF9tQVV1WVXurau/hw4dXqFwANhJBLQCsQbOMnze0e3pVdVXtXM36AOAUVYvM6wXTm5PsSPLkJJckeU1VPeJ+K3Vf0907u3vnli1b5l4oABuPoBYA1pip8fMuSHJ2kkuq6uxF2j0syfOTvHt1KwSAU9ahJGdNTZ+Z5PZF2vyX7v5Cdx9IcksmwS0ArChBLQCsPbOMn5ckP5/k5Uk+v5rFAcApbE+SHVW1vapOS3Jxkl0L2vx2kr+fJFV1eiZDIdy6qlUCsCEJagFg7Vl2/LyqOifJWd39u6tZGACcyrr7SJLLk9yQ5ANJ3tjdN1fVVVV14dDshiR3VNW+JO9I8lPdfcc4FQOwkWweuwAA4H6OOX5eVT0gyS8nefayG6q6LMllSbJ169Y5lQcAp67u3p1k94J5V0497iQvGH4AYNXoUQsAa89y4+c9LMnXJXlnVR1M8s1Jdi12QzE3OgEAADg1CGoBYO055vi7D+zrAAAYBUlEQVR53f2J7j69u7d197YkNya5sLv3jlMuAAAAJ0tQCwBrzIzj5wEAALCOGKMWANag5cbPWzD/yatREwAAACtHj1oAAAAAgJEJagEAAAAARiaoBQAAAAAY2UxBbVWdX1W3VNX+qrpikeUvqKp9VfX+qvr9qnrM1LJnVdUHh59nzbN4AAAAAID1YNmgtqo2Jbk6yQVJzk5ySVWdvaDZe5Ps7O5vSPLmJC8f1n1UkpckeVKSc5O8pKoeOb/yAQAAAABOfbP0qD03yf7uvrW770lyfZKLpht09zu6+7PD5I1JzhwePzXJ27r7zu6+K8nbkpw/n9IBAAAAANaHWYLaM5LcNjV9aJi3lEuTvPUE1wUAAAAA2HA2z9CmFpnXizasemaSnUm+/XjWrarLklyWJFu3bp2hJAAAAACA9WOWHrWHkpw1NX1mktsXNqqq85L8TJILu/vu41m3u6/p7p3dvXPLli2z1g4AAAAAsC7MEtTuSbKjqrZX1WlJLk6ya7pBVZ2T5NWZhLQfm1p0Q5LvqqpHDjcR+65hHgAAAAAAg2WHPujuI1V1eSYB66Yk13b3zVV1VZK93b0rySuSPDTJm6oqST7S3Rd2951V9fOZhL1JclV337kizwQAAAAA4BQ1yxi16e7dSXYvmHfl1OPzjrHutUmuPdECAQAAAADWu1mGPgAAAAAAYAUJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagFgDaqq86vqlqraX1VXLLL8eVX151V1U1W9q6rOHqNOAAAA5kNQCwBrTFVtSnJ1kguSnJ3kkkWC2Ou6++u7+wlJXp7klatcJgAAAHMkqAWAtefcJPu7+9buvifJ9Ukumm7Q3Z+cmnxIkl7F+gAAAJizzWMXAADczxlJbpuaPpTkSQsbVdUPJ3lBktOSfMfqlAYAAMBK0KMWANaeWmTe/XrMdvfV3f3YJC9M8uJFN1R1WVXtraq9hw8fnnOZAAAAzIugFgDWnkNJzpqaPjPJ7cdof32Spy22oLuv6e6d3b1zy5YtcywRAACAeRLUAsDasyfJjqraXlWnJbk4ya7pBlW1Y2rye5J8cBXrAwAAYM6MUQvM1fbtB0fb94ED20bbN8xTdx+pqsuT3JBkU5Jru/vmqroqyd7u3pXk8qo6L8kXktyV5FnjVQwAAMDJEtQCwBrU3buT7F4w78qpxz+66kUBAACwYmYa+qCqzq+qW6pqf1Vdscjyb6uqP6uqI1X19AXL7q2qm4afXQvXBQAAgNWy3PfbqXZPr6quqp2rWR8AG9eyPWqralOSq5M8JZObm+ypql3dvW+q2UeSPDvJTy6yic919xPmUCsAAACcsBm/36aqHpbk+UnevfpVArBRzdKj9twk+7v71u6+J5M7S1803aC7D3b3+5N8cQVqBAAAgHlY9vvt4OeTvDzJ51ezOAA2tlmC2jOS3DY1fWiYN6sHV9Xeqrqxqp62WIOqumxos/fw4cPHsWkAAACY2bLfb6vqnCRndffvrmZhADBLUFuLzOvj2MfW7t6Z5AeS/Juqeuz9NtZ9TXfv7O6dW7ZsOY5NAwAAwMyO+f22qh6Q5JeT/MSyG9LhCIA5myWoPZTkrKnpM5PcPusOuvv24d9bk7wzyTnHUR8AAADMy3Lfbx+W5OuSvLOqDib55iS7FruhmA5HAMzbLEHtniQ7qmp7VZ2W5OIku2bZeFU9sqoeNDw+Pcm3JNl37LUAAABgRRzz+213f6K7T+/ubd29LcmNSS7s7r3jlAvARrJsUNvdR5JcnuSGJB9I8sbuvrmqrqqqC5Okqp5YVYeSfF+SV1fVzcPqX5tkb1W9L8k7krxs4d00AQAAYDXM8v0WAMayeZZG3b07ye4F866cerwnk0tGFq73J0m+/iRrBAAAgLlY7vvtgvlPXo2aACCZbegDAAAAAABWkKAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAYmaAWAAAAAGBkgloAAAAAgJEJagEAAAAARiaoBQAAAAAY2eaxCwCOz/btB0fd/4ED20bdPwAAAMB6pEctAAAAAMDIBLUAAAAAACMT1AIAAAAAjExQCwAAAAAwMkEtAAAAAMDIBLUAAAAAACMT1ALAGlRV51fVLVW1v6quWGT5C6pqX1W9v6p+v6oeM0adAAAAzIegFgDWmKralOTqJBckOTvJJVV19oJm702ys7u/Icmbk7x8dasEAABgnjaPXQDAatm+/eCo+z9wYNuo++eUcm6S/d19a5JU1fVJLkqy72iD7n7HVPsbkzxzVSsEAABgrvSoBYC154wkt01NHxrmLeXSJG9dbEFVXVZVe6tq7+HDh+dYIgAAAPMkqAWAtacWmdeLNqx6ZpKdSV6x2PLuvqa7d3b3zi1btsyxRAAAAOZppqB2hhuafFtV/VlVHamqpy9Y9qyq+uDw86x5FQ4A69ihJGdNTZ+Z5PaFjarqvCQ/k+TC7r57lWoDAABgBSwb1M54Q5OPJHl2kusWrPuoJC9J8qRMxtt7SVU98uTLBoB1bU+SHVW1vapOS3Jxkl3TDarqnCSvziSk/dgINQIAADBHs/So/esbmnT3PUmO3tDkr3X3we5+f5IvLlj3qUne1t13dvddSd6W5Pw51A0A61Z3H0lyeZIbknwgyRu7++aquqqqLhyavSLJQ5O8qapuqqpdS2wOAACAU8DmGdosdkOTJ824/eO9GQoAkKS7dyfZvWDelVOPz1v1ogAAAFgxs/SonfmGJie6rjtSAwAAAAAb2SxB7Uw3NDmZdd2RGgAAAADYyGYJape9ockx3JDku6rqkcNNxL5rmAcAAAAAwGDZoHaWG5pU1ROr6lCS70vy6qq6eVj3ziQ/n0nYuyfJVcM8AAAAAAAGs9xMbJYbmuzJZFiDxda9Nsm1J1EjAAAAAMC6NsvQBwAAAAAArCBBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAABtGVZ1fVbdU1f6qumKR5S+oqn1V9f6q+v2qeswYdQKw8QhqAQAA2BCqalOSq5NckOTsJJdU1dkLmr03yc7u/oYkb07y8tWtEoCNSlALAADARnFukv3dfWt335Pk+iQXTTfo7nd092eHyRuTnLnKNQKwQQlqAQAA2CjOSHLb1PShYd5SLk3y1hWtCAAGm8cuAAAAAFZJLTKvF21Y9cwkO5N8+xLLL0tyWZJs3bp1XvUBsIHpUQsAAMBGcSjJWVPTZya5fWGjqjovyc8kubC7715sQ919TXfv7O6dW7ZsWZFiAdhYBLUAAABsFHuS7Kiq7VV1WpKLk+yablBV5yR5dSYh7cdGqBGADUpQCwAAwIbQ3UeSXJ7khiQfSPLG7r65qq6qqguHZq9I8tAkb6qqm6pq1xKbA4C5MkYtAAAAG0Z3706ye8G8K6cen7fqRQFABLUAALDmHXzt9lH3v+3SA6PuHwBgIzD0AQAAAADAyAS1AAAAAAAjE9QCAAAAAIzMGLWMZvv2g6Pt+8CBbaPtGwAAAAAW0qMWANagqjq/qm6pqv1VdcUiy7+tqv6sqo5U1dPHqBEAAID5EdQCwBpTVZuSXJ3kgiRnJ7mkqs5e0OwjSZ6d5LrVrQ4AAICVYOgDAFh7zk2yv7tvTZKquj7JRUn2HW3Q3QeHZV8co0AAAADma6YetTNcfvmgqvrNYfm7q2rbMH9bVX2uqm4afl413/IBYF06I8ltU9OHhnkAAACsU8v2qJ26/PIpmXxR3FNVu7p731SzS5Pc1d1fXVUXJ/mlJN8/LPtQdz9hznUDwHpWi8zrE9pQ1WVJLkuSrVu3nkxNAAAArKBZhj5Y9vLLYfqlw+M3J/mVqlrsS+Zx27794Dw2c8IOHNg26v4B2JAOJTlravrMJLefyIa6+5ok1yTJzp07TyjsBQAAYOXNMvTBLJdf/nWb7j6S5BNJvnxYtr2q3ltVf1BV33qS9QLARrAnyY6q2l5VpyW5OMmukWsCAABgBc0S1M5y+eVSbT6aZGt3n5PkBUmuq6qH328HVZdV1d6q2nv48OEZSgKA9Wv4o+flSW5I8oEkb+zum6vqqqq6MEmq6olVdSjJ9yV5dVXdPF7FAAAAnKxZhj6Y5fLLo20OVdXmJF+W5M7u7iR3J0l3v6eqPpTkcUn2Tq/sskwAuK/u3p1k94J5V0493pPJORkAAIB1YJYetbNcfrkrybOGx09P8vbu7qraMtyMLFX1VUl2JLl1PqUDAAAAAKwPy/ao7e4jVXX08stNSa49evllkr3dvSvJa5O8vqr2J7kzkzA3Sb4tyVVVdSTJvUme1913rsQTAQAAAAA4Vc0y9MEsl19+PpMx8hau95YkbznJGgHWve3bD466/wMHto26fwAAANjoZhn6AAAAAACAFTRTj1oAAFjvDr52+2j73nbpgdH2DQDA2qBHLQAAAADAyAS1AAAAAAAjM/TBSRrzBkBu/gMAAAAA64MetQAAAAAAIxPUAgAAAACMzNAHAADASTn42u2j7XvbpQdG2zcAwDzpUQsAAAAAMDJBLQAAAADAyAS1AAAAAAAjE9QCAAAAAIxMUAsAAAAAMDJBLQAAAADAyDaPXQAAa9/27QdH2/eBA9tG2zcAAACsFj1qAQAAAABGJqgFAAAAABiZoQ/WsTEvVU5crgwAAAAAs9KjFgAAAABgZIJaAAAAAICRCWoBAAAAAEYmqAUAAAAAGJmgFgAAAABgZJvHLgAAgPk5+Nrto+5/26UHRt0/AACcqvSoBQAAAAAYmR61ALBB6XkJAACwdswU1FbV+Un+bZJNSV7T3S9bsPxBSX49yTcluSPJ93f3wWHZi5JcmuTeJM/v7hvmVj0ArFMnc+4F4G/4oxQLOccCsFYtG9RW1aYkVyd5SpJDSfZU1a7u3jfV7NIkd3X3V1fVxUl+Kcn3V9XZSS5O8vgkj07ye1X1uO6+d95PBADWi5M5965+tStDsALASnCOBWAtm2WM2nOT7O/uW7v7niTXJ7loQZuLkrxuePzmJN9ZVTXMv7677+7uA0n2D9sDAJZ2MudeAGBpzrEArFmzDH1wRpLbpqYPJXnSUm26+0hVfSLJlw/zb1yw7hknXC0AbAwnc+79+KpUuMGN2eNXb19YX/w+WXXOsQCsWbMEtYv95bBnbDPLuqmqy5JcNkx+uqpumaGuWZ2ekzihrsLfTU+4vrVcW7Li9a3l2hKv64lay7Ula7u+tVxbMt/PxGNOtphTwMmce+/baA2fY/OcFX3jreXakpOpby3Xlnhdva4nyrE7MfOuzTl29jYreY49udd15a3l+tZybcnaru/kalvZLxRr+bgla7u+tVxbsrbrm3dtM51jZwlqDyU5a2r6zCS3L9HmUFVtTvJlSe6ccd109zVJrpml4ONVVXu7e+dKbHse1nJ9ajtxa7k+tZ24tVzfWq4tWfv1rUEnc+69j416jl3LtSVruz61nbi1XN9ari1Z2/Wpbd1Z8+fYtf66ruX61nJtyf/f3t2F7F3XcRx/f9o03UoMe2BthguGFB7kkPUwGNEytMQeKJhQRCdFzJh1ENVBUeBBENFZENvKyia2KQwZPoA9Hvi0qbg5q2Wmt1ozqGw9MGefDv7/rWs3rnb//9t+3+vi84KxezcOP17bfb/lf13/31V7X7YNV3lf5W1Qe1+rbSdzRu39wCpJKyWdTffmYDvn/TM7gY/3H38YuNu2+89vkPRySSuBVcB9p2Z6RETEzBrT3oiIiDixNDYiIsr6v6+o7c/kuRa4A1gEbLW9T9LXgAds7wS2AD+QdIDumcYN/e/dJ+lm4FHgCLDR9oun6b8lIiJiJoxpb0RERJxYGhsREZWdzNEH2N4F7Jr3uS9PfPwv4CMn+L3XA9eP2DjWabnd8xSqvC/bhqu8L9uGq7yv8jaov6+cMe09gyr/uVbeBrX3ZdtwlfdV3ga192XbjJmCxlb/c628r/I2qL0v24arvK/yNqi9r8k25Q6OiIiIiIiIiIiIiLZO5ozaiIiIiIiIiIiIiDiNZvpCraQrJP1K0gFJX2i95yhJWyUdlLS39Zb5JF0o6SeS9kvaJ2lT602TJJ0j6T5JD/f7vtp603ySFkl6UNJtrbfMJ+kJSY9IekjSA633TJJ0vqTtkh7r//69vfUmAEkX94/X0R/PS7qu9a5Jkj7bfz3slbRN0jmtNx0laVO/a1+1xy2Gq9pXSGPHSGOHq9xXSGOHqtxXSGNnVRo7TOXGpq/jVG5s1b5CGjtG677O7NEHkhYBvwYuB+bo3t3zGtuPNh0GSFoHHAK+b/uS1nsmSVoGLLO9R9Irgd3AByo8bgCSBCy1fUjSWcAvgU2272k87RhJnwMuA86zfVXrPZMkPQFcZvtPrbfMJ+kG4Be2N6t7B94ltv/Setek/vvK08Bbbf++9R4AScvpvg7ebPuf6t7AcZft77VdBpIuAW4C1gCHgduBT9v+TdNhMUrlvkIaO0YaO1zlvkIaO3BP2b5CGjur0tjhKjc2fR2ncmOnoa+Qxi5Ehb7O8itq1wAHbD9u+zDdA/3+xpsAsP1zuncPLcf2s7b39B//DdgPLG+76r/cOdT/8qz+R5lnGyStAN4HbG69ZZpIOg9YR/cOu9g+XDFwwHrgtxXiNs9i4FxJi4ElwDON9xz1JuAe2/+wfQT4GfDBxptivLJ9hTR2jDR2NqWxo1TtK6SxsyqNHahyY9PX2TRFfYU0diGa93WWL9QuB56a+PUcRb5RTwtJFwGXAve2XXK8/raMh4CDwF22K+37FvB54N+th5yAgTsl7Zb0ydZjJrwReA74bn/LzWZJS1uPegkbgG2tR0yy/TTwDeBJ4Fngr7bvbLvqmL3AOkkXSFoCvBe4sPGmGC99PQXS2EEqN7ZqXyGNHaR4XyGNnVVp7ClQsbHp6yhVGzstfYU0diGa93WWL9TqJT5X5lmr6iS9AtgBXGf7+dZ7Jtl+0fZbgBXAmv6l6c1Jugo4aHt36y3/w1rbq4ErgY397UsVLAZWA9+2fSnwd6DamVxnA1cDP269ZZKkV9G90mIl8HpgqaSPtl3Vsb0f+DpwF90tIw8DR5qOilMhfR0pjV24KWhs1b5CGjtI5b5CGjvD0tiRqjY2fR2lamPL9xXS2IWq0NdZvlA7x/FXvVdQ56XUpfXn5uwAbrR9S+s9J9LfVvBT4IrGU45aC1zdn6FzE/AuST9sO+l4tp/pfz4I3Ep3e1UFc8DcxDPL2+miV8mVwB7bf2w9ZJ53A7+z/ZztF4BbgHc03nSM7S22V9teR3erXM7Om37p6whp7GClG1u4r5DGDlW6r5DGzqg0doRpaGz6unCFGzsNfYU0dsFa93WWL9TeD6yStLJ/BmEDsLPxpvL6g863APttf7P1nvkkvUbS+f3H59J9gT/WdlXH9hdtr7B9Ed3ft7ttl3hWCEDS0v5gffpbMt5D97L+5mz/AXhK0sX9p9YDzQ/+n+caCt0uMuFJ4G2SlvRfv+vpzuQqQdJr+5/fAHyImo9hLEz6OlAaO1zlxlbuK6SxI5TuK6SxMyqNHahyY9PX4So3dkr6CmnsgrXu6+Iz+S87k2wfkXQtcAewCNhqe1/jWQBI2ga8E3i1pDngK7a3tF11zFrgY8Aj/Rk6AF+yvavhpknLgBvUvWvhy4Cbbd/WeNO0eB1wa/d9kMXAj2zf3nbScT4D3Nj/T+njwCca7zmmP5vmcuBTrbfMZ/teSduBPXS3ZDwIfKftquPskHQB8AKw0fafWw+KcSr3FdLYkdLYYar3FdLYBZuCvkIaO3PS2FEqNzZ9Ha56Y8v2FdLYEZr2VXaOvImIiIiIiIiIiIhoaZaPPoiIiIiIiIiIiIiYCrlQGxEREREREREREdFYLtRGRERERERERERENJYLtRERERERERERERGN5UJtRERERERERERERGO5UBsRERERERERERHRWC7URkRERERERERERDSWC7URERERERERERERjf0HXwoGmceicHEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1728x360 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "a = torch.softmax(torch.randn(10),0)\n",
    "fig, axes = plt.subplots(1,3,figsize=(24,5))\n",
    "fig.suptitle(\"The effect of sharpening on randomly generated distribution\")\n",
    "sns.barplot(x=np.arange(10),y=a,color=\"blue\",ax=axes[0])\n",
    "axes[0].set_title(\"$T = 1.0$\")\n",
    "sns.barplot(x=np.arange(10),y=sharpen(a[None,:],0.5)[0],color=\"orange\",ax=axes[1])\n",
    "axes[1].set_title(\"$T = 0.5$\")\n",
    "sns.barplot(x=np.arange(10),y=sharpen(a[None,:],0.1)[0],color=\"red\",ax=axes[2])\n",
    "axes[2].set_title(\"$T = 0.1$\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Mixmatch Algorithm\n",
    "\n",
    "Now with all the pieces in place, the full algorithm can be implemented. Here are the steps for a single training iteration:\n",
    "\n",
    "1. Supply a batch of labeled data with its labels, and a batch of unlabeled data\n",
    "2. Augment the labeled batch to produce a new training batch. \n",
    "3. Augment each image in the unlabeled batch $K$ times, to produce a total of  $BatchSize * K$ new unlabeled examples.\n",
    "4. For each original image in the unlabeled batch, pass the $K$ augmented versions to the model. Average the model's predictions across the augments to produce a single pseudo-label for the augmented images. \n",
    "5. Sharpen the pseudo-labels. \n",
    "6. The augmented labeled dataset and it's labels form set $X$. The augmented unlabeled data and it's (predicted) labels form set $U$.\n",
    "7. Concatenate sets $U$ and $X$ into set $W$. Shuffle $W$. \n",
    "8. Form set $X'$ by applying mixup to sets $X$ and $|X|$ examples from $W$.\n",
    "9. Form set $U'$ by applying mixup to sets $U$ and the examples in $W$ that were not used in step 8.\n",
    "\n",
    "Sets $X'$ (labeled mixup) and $U'$ (unlabeled mixup) are then passed to the model, and the loss is computed using the corresponding mixed-up labels. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Model\n",
    "\n",
    "We will use a wideresnet model with 28 layers and a growth factor of 2 to match the paper. I use fastai's included WRN implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = models.WideResNet(num_groups=3,N=4,num_classes=10,k=2,start_nf=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss\n",
    "\n",
    "With data and model in hand, we'll now implement the final piece requried for training -- the loss function. The loss function is the summation of two terms; the labeled and unlabeled losses. The labeled loss uses standard cross entropy, however the unlabeled loss function is the $l_2$ loss instead. This is because the $l_2$ loss is much less sensitive to very incorrect predicitions. Cross entropy loss is unbounded, and as the model's predicted probability of the correct class goes to zero cross entropy goes to infinty. However with $l_2$ loss, since we are working with probabilities, the worst case is that the model predicts 0 when the target is 1 or vice versa; this results in a loss of 1. With the unlabeled targets coming from the model itself, the algorithm doesn't want to penalize incorrect predictions too harshly. The parameter $\\lambda$ (`l` in the code since lambda is reserved) controls the balance between the two terms.\n",
    "\n",
    "We'll make one slight departure from the paper by linearly ramping up the weight of the unlabeled loss over the first 3000 iterations (roughly 10 epochs). Before applying this rampup, I was having difficulty training the model and found the accuracy would increase very slowly at in early epochs. Since the predicted labels at the start of training are essentially random, it makes sense to delay the application of unlabeled loss. By the time the weight of the unlabeled loss becomes significant, the model should be making reasonably good predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixupLoss(nn.Module):\n",
    "    def forward(self, preds, target, unsort=None, ramp=None, bs=None):\n",
    "        if unsort is None:\n",
    "            return F.cross_entropy(preds,target)\n",
    "        preds = preds[unsort]\n",
    "        preds_l = preds[:bs]\n",
    "        preds_ul = preds[bs:]\n",
    "        preds_l = torch.log_softmax(preds_l,dim=1)\n",
    "        preds_ul = torch.softmax(preds_ul,dim=1)\n",
    "        loss_x = -(preds_l * target[:bs]).sum(dim=1).mean()\n",
    "        loss_u = F.mse_loss(preds_ul,target[bs:])\n",
    "        self.loss_x = loss_x.item()\n",
    "        self.loss_u = loss_u.item()\n",
    "        return loss_x + 100 * ramp * loss_u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Before training, lets review the hyperparemeters that have been introduced. \n",
    "\n",
    "| Hyperparameter | Description | Value |\n",
    "| -------------- | ----------- | ----- |\n",
    "| $K$            | Number of augments | 2 |\n",
    "| $T$ | Sharpening Temperature | 0.5 |\n",
    "| $\\alpha$ | Beta dist. parameter | 0.75 |\n",
    "| $\\lambda$ | Unlabeled loss weight | 75 |\n",
    "\n",
    "The authors of the paper claim that $T$ and $K$ should be relatively constant across most datasets, while $\\alpha$ and $\\lambda$ need to be tuned per set. We'll use the same hyperparameters as the paper's [official implementation](https://github.com/google-research/mixmatch). \n",
    "\n",
    "One implementation detail: the paper mentions that instead of learning rate annealing, it updates a second model with the exponentially moving average of the training model's parameters. This is yet another form of regularization, but is not essential to the algorithm. For those interested, there is code for training with an EMA model in the repository. However, I didn't find a significant benefit over learning rate scheduling, and in the name of simplicity we'll forgo EMA and use fastai's implementaion of the [one cycle policy](https://docs.fast.ai/callbacks.one_cycle.html#What-is-1cycle?) to schedule the learning and momentum rates. \n",
    "\n",
    "We'll use fastai's callback system to write a method which handles most of the MixMatch steps. This method takes in batches from the labeled and unlabeled sets, gets the predicted labels, and then performs mixup. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixMatchTrainer(LearnerCallback):\n",
    "    _order=-20\n",
    "    def on_train_begin(self, **kwargs):\n",
    "        self.l_dl = iter(data_labeled.train_dl)\n",
    "        self.smoothL, self.smoothUL = SmoothenValue(0.98), SmoothenValue(0.98)\n",
    "        self.recorder.add_metric_names([\"l_loss\",\"ul_loss\"])\n",
    "        self.it = 0\n",
    "        \n",
    "    def on_batch_begin(self, train, last_input, last_target, **kwargs):\n",
    "        if not train: return\n",
    "        try:\n",
    "            x_l,y_l = next(self.l_dl)\n",
    "        except:\n",
    "            self.l_dl = iter(data_labeled.train_dl)\n",
    "            x_l,y_l = next(self.l_dl)\n",
    "            \n",
    "        x_ul = last_input\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            ul_labels = sharpen(torch.softmax(torch.stack([self.learn.model(x_ul[:,i]) for i in range(x_ul.shape[1])],dim=1),dim=2).mean(dim=1))\n",
    "            \n",
    "        x_ul = torch.cat([x for x in x_ul])\n",
    "        ul_labels = torch.cat([y.unsqueeze(0).expand(K,-1) for y in ul_labels])\n",
    "        \n",
    "        l_labels = torch.eye(data_labeled.c).cuda()[y_l]\n",
    "        \n",
    "        w_x = torch.cat([x_l,x_ul])\n",
    "        w_y = torch.cat([l_labels,ul_labels])\n",
    "        idxs = torch.randperm(w_x.shape[0])\n",
    "        \n",
    "        mixed_input, mixed_target = mixup(w_x,w_y,w_x[idxs],w_y[idxs])\n",
    "        bn_idxs = torch.randperm(mixed_input.shape[0])\n",
    "        unsort = [0] * len(bn_idxs)\n",
    "        for i,j in enumerate(bn_idxs): unsort[j] = i\n",
    "        mixed_input = mixed_input[bn_idxs]\n",
    "    \n",
    "\n",
    "        ramp = self.it / 3000.0 if self.it < 3000 else 1.0\n",
    "        return {\"last_input\": mixed_input, \"last_target\": (mixed_target,unsort,ramp,x_l.shape[0])}\n",
    "    \n",
    "    def on_batch_end(self, train, **kwargs):\n",
    "        if not train: return\n",
    "        self.smoothL.add_value(self.learn.loss_func.loss_x)\n",
    "        self.smoothUL.add_value(self.learn.loss_func.loss_u)\n",
    "        self.it += 1\n",
    "\n",
    "    def on_epoch_end(self, last_metrics, **kwargs):\n",
    "        return add_metrics(last_metrics,[self.smoothL.smooth,self.smoothUL.smooth])    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A fastai `Learner` object contains the dataloaders and the model, and is responsible for executing the training loop. It also has a lot of utility functions, such as learning rate finding and prediction interpretation. An epoch in this implementation is one pass through the entire unlabeled dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data_unlabeled,model,loss_func=MixupLoss(),callback_fns=[MixMatchTrainer],metrics=accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reference, I ran these tests on a Google Compute Engine virtual machine with 16 CPUs and a single P100 GPU. The first step is to establish some baselines so that MixMatch's performance can be compared. First, I'll try the fully supervised case with all 50k training images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 3:14:40 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>1.191859</th>\n",
       "    <th>1.232437</th>\n",
       "    <th>0.570300</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.923196</th>\n",
       "    <th>0.926058</th>\n",
       "    <th>0.676400</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.764989</th>\n",
       "    <th>0.983374</th>\n",
       "    <th>0.688000</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.684145</th>\n",
       "    <th>0.792043</th>\n",
       "    <th>0.742400</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>0.637452</th>\n",
       "    <th>0.706040</th>\n",
       "    <th>0.763000</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>0.579363</th>\n",
       "    <th>0.792375</th>\n",
       "    <th>0.738900</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>0.560745</th>\n",
       "    <th>0.761193</th>\n",
       "    <th>0.748400</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>0.542322</th>\n",
       "    <th>0.623887</th>\n",
       "    <th>0.793000</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>0.505341</th>\n",
       "    <th>0.625939</th>\n",
       "    <th>0.797400</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>0.478072</th>\n",
       "    <th>0.605989</th>\n",
       "    <th>0.810400</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>0.450396</th>\n",
       "    <th>0.534871</th>\n",
       "    <th>0.823500</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>0.456539</th>\n",
       "    <th>0.493163</th>\n",
       "    <th>0.834700</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>0.428605</th>\n",
       "    <th>0.489522</th>\n",
       "    <th>0.838600</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>0.410763</th>\n",
       "    <th>0.489317</th>\n",
       "    <th>0.837100</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>0.392021</th>\n",
       "    <th>0.499354</th>\n",
       "    <th>0.839900</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>0.388261</th>\n",
       "    <th>0.484821</th>\n",
       "    <th>0.843200</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>0.383759</th>\n",
       "    <th>0.478408</th>\n",
       "    <th>0.848900</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>0.359334</th>\n",
       "    <th>0.420070</th>\n",
       "    <th>0.865200</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>0.341625</th>\n",
       "    <th>0.481819</th>\n",
       "    <th>0.842300</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>0.347987</th>\n",
       "    <th>0.531025</th>\n",
       "    <th>0.835500</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>0.334328</th>\n",
       "    <th>0.459328</th>\n",
       "    <th>0.851300</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>0.308619</th>\n",
       "    <th>0.417754</th>\n",
       "    <th>0.862000</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>0.299601</th>\n",
       "    <th>0.429223</th>\n",
       "    <th>0.862100</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>0.290227</th>\n",
       "    <th>0.407856</th>\n",
       "    <th>0.863300</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>0.295873</th>\n",
       "    <th>0.453121</th>\n",
       "    <th>0.859800</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>0.258007</th>\n",
       "    <th>0.351765</th>\n",
       "    <th>0.885200</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>0.255791</th>\n",
       "    <th>0.414505</th>\n",
       "    <th>0.873400</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>0.236708</th>\n",
       "    <th>0.369134</th>\n",
       "    <th>0.883600</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>0.240606</th>\n",
       "    <th>0.340578</th>\n",
       "    <th>0.890700</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>0.230230</th>\n",
       "    <th>0.316679</th>\n",
       "    <th>0.902600</th>\n",
       "    <th>01:52</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>0.213636</th>\n",
       "    <th>0.422995</th>\n",
       "    <th>0.873600</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>31</th>\n",
       "    <th>0.217762</th>\n",
       "    <th>0.409919</th>\n",
       "    <th>0.879000</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>32</th>\n",
       "    <th>0.193347</th>\n",
       "    <th>0.368671</th>\n",
       "    <th>0.888600</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>33</th>\n",
       "    <th>0.187700</th>\n",
       "    <th>0.303848</th>\n",
       "    <th>0.907300</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>34</th>\n",
       "    <th>0.178209</th>\n",
       "    <th>0.328777</th>\n",
       "    <th>0.901500</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>35</th>\n",
       "    <th>0.173358</th>\n",
       "    <th>0.317374</th>\n",
       "    <th>0.902900</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>36</th>\n",
       "    <th>0.166649</th>\n",
       "    <th>0.304963</th>\n",
       "    <th>0.904500</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>37</th>\n",
       "    <th>0.151209</th>\n",
       "    <th>0.283942</th>\n",
       "    <th>0.911800</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>38</th>\n",
       "    <th>0.160273</th>\n",
       "    <th>0.264917</th>\n",
       "    <th>0.919200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>39</th>\n",
       "    <th>0.140941</th>\n",
       "    <th>0.266427</th>\n",
       "    <th>0.921000</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>40</th>\n",
       "    <th>0.135635</th>\n",
       "    <th>0.300873</th>\n",
       "    <th>0.916400</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>41</th>\n",
       "    <th>0.132049</th>\n",
       "    <th>0.290825</th>\n",
       "    <th>0.918200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>42</th>\n",
       "    <th>0.126680</th>\n",
       "    <th>0.277839</th>\n",
       "    <th>0.919100</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>43</th>\n",
       "    <th>0.117725</th>\n",
       "    <th>0.270094</th>\n",
       "    <th>0.921600</th>\n",
       "    <th>01:56</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>44</th>\n",
       "    <th>0.125699</th>\n",
       "    <th>0.262170</th>\n",
       "    <th>0.922900</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>45</th>\n",
       "    <th>0.110507</th>\n",
       "    <th>0.285798</th>\n",
       "    <th>0.919900</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>46</th>\n",
       "    <th>0.108991</th>\n",
       "    <th>0.290960</th>\n",
       "    <th>0.919000</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>47</th>\n",
       "    <th>0.101317</th>\n",
       "    <th>0.279039</th>\n",
       "    <th>0.922500</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>48</th>\n",
       "    <th>0.091395</th>\n",
       "    <th>0.288884</th>\n",
       "    <th>0.923700</th>\n",
       "    <th>01:53</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>49</th>\n",
       "    <th>0.094465</th>\n",
       "    <th>0.261242</th>\n",
       "    <th>0.927000</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>50</th>\n",
       "    <th>0.084766</th>\n",
       "    <th>0.340163</th>\n",
       "    <th>0.914600</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>51</th>\n",
       "    <th>0.082190</th>\n",
       "    <th>0.269969</th>\n",
       "    <th>0.929200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>52</th>\n",
       "    <th>0.074630</th>\n",
       "    <th>0.304642</th>\n",
       "    <th>0.921500</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>53</th>\n",
       "    <th>0.078128</th>\n",
       "    <th>0.252954</th>\n",
       "    <th>0.932200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>54</th>\n",
       "    <th>0.075592</th>\n",
       "    <th>0.270782</th>\n",
       "    <th>0.925200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>55</th>\n",
       "    <th>0.068515</th>\n",
       "    <th>0.261388</th>\n",
       "    <th>0.932100</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>56</th>\n",
       "    <th>0.056724</th>\n",
       "    <th>0.305182</th>\n",
       "    <th>0.926900</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>57</th>\n",
       "    <th>0.065933</th>\n",
       "    <th>0.258175</th>\n",
       "    <th>0.934400</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>58</th>\n",
       "    <th>0.060691</th>\n",
       "    <th>0.268944</th>\n",
       "    <th>0.932300</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>59</th>\n",
       "    <th>0.056501</th>\n",
       "    <th>0.251997</th>\n",
       "    <th>0.935400</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>60</th>\n",
       "    <th>0.059016</th>\n",
       "    <th>0.253136</th>\n",
       "    <th>0.934200</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>61</th>\n",
       "    <th>0.050735</th>\n",
       "    <th>0.247101</th>\n",
       "    <th>0.935800</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>62</th>\n",
       "    <th>0.043475</th>\n",
       "    <th>0.268170</th>\n",
       "    <th>0.935700</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>63</th>\n",
       "    <th>0.036704</th>\n",
       "    <th>0.262504</th>\n",
       "    <th>0.938600</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>64</th>\n",
       "    <th>0.041342</th>\n",
       "    <th>0.260229</th>\n",
       "    <th>0.935800</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>65</th>\n",
       "    <th>0.038287</th>\n",
       "    <th>0.267951</th>\n",
       "    <th>0.940200</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>66</th>\n",
       "    <th>0.036856</th>\n",
       "    <th>0.264761</th>\n",
       "    <th>0.939200</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>67</th>\n",
       "    <th>0.032310</th>\n",
       "    <th>0.255510</th>\n",
       "    <th>0.941900</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>68</th>\n",
       "    <th>0.030962</th>\n",
       "    <th>0.261424</th>\n",
       "    <th>0.939400</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>69</th>\n",
       "    <th>0.030238</th>\n",
       "    <th>0.278251</th>\n",
       "    <th>0.939000</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>70</th>\n",
       "    <th>0.027311</th>\n",
       "    <th>0.258946</th>\n",
       "    <th>0.940400</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>71</th>\n",
       "    <th>0.024804</th>\n",
       "    <th>0.304373</th>\n",
       "    <th>0.936700</th>\n",
       "    <th>01:59</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>72</th>\n",
       "    <th>0.022955</th>\n",
       "    <th>0.291469</th>\n",
       "    <th>0.940000</th>\n",
       "    <th>01:59</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>73</th>\n",
       "    <th>0.020202</th>\n",
       "    <th>0.256995</th>\n",
       "    <th>0.942200</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>74</th>\n",
       "    <th>0.020392</th>\n",
       "    <th>0.276180</th>\n",
       "    <th>0.940400</th>\n",
       "    <th>01:59</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>75</th>\n",
       "    <th>0.020351</th>\n",
       "    <th>0.289667</th>\n",
       "    <th>0.939000</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>76</th>\n",
       "    <th>0.017046</th>\n",
       "    <th>0.269220</th>\n",
       "    <th>0.941200</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>77</th>\n",
       "    <th>0.013297</th>\n",
       "    <th>0.278037</th>\n",
       "    <th>0.945300</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>78</th>\n",
       "    <th>0.014478</th>\n",
       "    <th>0.277517</th>\n",
       "    <th>0.946100</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>79</th>\n",
       "    <th>0.014263</th>\n",
       "    <th>0.303503</th>\n",
       "    <th>0.941700</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>80</th>\n",
       "    <th>0.010069</th>\n",
       "    <th>0.277407</th>\n",
       "    <th>0.946300</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>81</th>\n",
       "    <th>0.010061</th>\n",
       "    <th>0.274015</th>\n",
       "    <th>0.945600</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>82</th>\n",
       "    <th>0.010733</th>\n",
       "    <th>0.291417</th>\n",
       "    <th>0.945600</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>83</th>\n",
       "    <th>0.012564</th>\n",
       "    <th>0.286028</th>\n",
       "    <th>0.945000</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>84</th>\n",
       "    <th>0.009532</th>\n",
       "    <th>0.276559</th>\n",
       "    <th>0.948100</th>\n",
       "    <th>01:55</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>85</th>\n",
       "    <th>0.010781</th>\n",
       "    <th>0.285107</th>\n",
       "    <th>0.946900</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>86</th>\n",
       "    <th>0.007245</th>\n",
       "    <th>0.287036</th>\n",
       "    <th>0.946000</th>\n",
       "    <th>01:54</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>87</th>\n",
       "    <th>0.006437</th>\n",
       "    <th>0.284886</th>\n",
       "    <th>0.945400</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>88</th>\n",
       "    <th>0.006796</th>\n",
       "    <th>0.284096</th>\n",
       "    <th>0.946400</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>89</th>\n",
       "    <th>0.006250</th>\n",
       "    <th>0.294242</th>\n",
       "    <th>0.945300</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>90</th>\n",
       "    <th>0.006399</th>\n",
       "    <th>0.291950</th>\n",
       "    <th>0.946800</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>91</th>\n",
       "    <th>0.003978</th>\n",
       "    <th>0.290678</th>\n",
       "    <th>0.947000</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>92</th>\n",
       "    <th>0.004333</th>\n",
       "    <th>0.285396</th>\n",
       "    <th>0.948100</th>\n",
       "    <th>01:57</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>93</th>\n",
       "    <th>0.003898</th>\n",
       "    <th>0.294682</th>\n",
       "    <th>0.947500</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>94</th>\n",
       "    <th>0.005112</th>\n",
       "    <th>0.290633</th>\n",
       "    <th>0.948200</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>95</th>\n",
       "    <th>0.005241</th>\n",
       "    <th>0.291476</th>\n",
       "    <th>0.948300</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>96</th>\n",
       "    <th>0.004306</th>\n",
       "    <th>0.289331</th>\n",
       "    <th>0.949100</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>97</th>\n",
       "    <th>0.004032</th>\n",
       "    <th>0.294601</th>\n",
       "    <th>0.948500</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>98</th>\n",
       "    <th>0.005866</th>\n",
       "    <th>0.288226</th>\n",
       "    <th>0.948500</th>\n",
       "    <th>01:58</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>99</th>\n",
       "    <th>0.004624</th>\n",
       "    <th>0.289931</th>\n",
       "    <th>0.947600</th>\n",
       "    <th>01:59</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learnFS = Learner(data_full,models.WideResNet(num_groups=3,N=4,num_classes=10,k=2,start_nf=32),metrics=accuracy)\n",
    "learnFS.fit_one_cycle(100,2e-2,wd=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I will try training on just the 500 labeled images, with no unsupervised component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 07:02 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>2.211227</th>\n",
       "    <th>2.303742</th>\n",
       "    <th>0.100000</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.136889</th>\n",
       "    <th>2.358332</th>\n",
       "    <th>0.100000</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.065619</th>\n",
       "    <th>3.168624</th>\n",
       "    <th>0.100500</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>2.010923</th>\n",
       "    <th>2.863336</th>\n",
       "    <th>0.215300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>1.961602</th>\n",
       "    <th>48.241714</th>\n",
       "    <th>0.107600</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.931448</th>\n",
       "    <th>7.775838</th>\n",
       "    <th>0.187700</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>1.920944</th>\n",
       "    <th>57.763363</th>\n",
       "    <th>0.102900</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>1.911295</th>\n",
       "    <th>6.584240</th>\n",
       "    <th>0.154000</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>1.898177</th>\n",
       "    <th>91.394135</th>\n",
       "    <th>0.106300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>1.898196</th>\n",
       "    <th>5.922698</th>\n",
       "    <th>0.213500</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>1.912254</th>\n",
       "    <th>13.457310</th>\n",
       "    <th>0.144800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>1.919444</th>\n",
       "    <th>23.014828</th>\n",
       "    <th>0.129200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>1.913109</th>\n",
       "    <th>17.263031</th>\n",
       "    <th>0.134600</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>1.909522</th>\n",
       "    <th>3.278674</th>\n",
       "    <th>0.171300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>1.904516</th>\n",
       "    <th>2.547180</th>\n",
       "    <th>0.241100</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>1.894310</th>\n",
       "    <th>5.688880</th>\n",
       "    <th>0.192200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>1.885931</th>\n",
       "    <th>2.265847</th>\n",
       "    <th>0.284700</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>1.866624</th>\n",
       "    <th>3.873827</th>\n",
       "    <th>0.211200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>1.859430</th>\n",
       "    <th>4.673414</th>\n",
       "    <th>0.208100</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.855025</th>\n",
       "    <th>3.200423</th>\n",
       "    <th>0.209100</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>1.858376</th>\n",
       "    <th>4.687626</th>\n",
       "    <th>0.197300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>1.845212</th>\n",
       "    <th>2.362020</th>\n",
       "    <th>0.235400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>1.831409</th>\n",
       "    <th>2.864459</th>\n",
       "    <th>0.214800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>1.827730</th>\n",
       "    <th>2.040632</th>\n",
       "    <th>0.281900</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>1.812962</th>\n",
       "    <th>2.694865</th>\n",
       "    <th>0.273800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>1.794275</th>\n",
       "    <th>1.849062</th>\n",
       "    <th>0.337500</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>1.770045</th>\n",
       "    <th>1.909112</th>\n",
       "    <th>0.310400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>1.749784</th>\n",
       "    <th>1.859538</th>\n",
       "    <th>0.329400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>1.730064</th>\n",
       "    <th>2.103187</th>\n",
       "    <th>0.290800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>1.708194</th>\n",
       "    <th>2.484337</th>\n",
       "    <th>0.274100</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>1.684061</th>\n",
       "    <th>2.292233</th>\n",
       "    <th>0.310400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>31</th>\n",
       "    <th>1.665761</th>\n",
       "    <th>2.060343</th>\n",
       "    <th>0.305400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>32</th>\n",
       "    <th>1.647016</th>\n",
       "    <th>1.837741</th>\n",
       "    <th>0.349300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>33</th>\n",
       "    <th>1.627093</th>\n",
       "    <th>1.925992</th>\n",
       "    <th>0.325700</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>34</th>\n",
       "    <th>1.605543</th>\n",
       "    <th>2.184734</th>\n",
       "    <th>0.321100</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>35</th>\n",
       "    <th>1.583871</th>\n",
       "    <th>2.129502</th>\n",
       "    <th>0.326600</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>36</th>\n",
       "    <th>1.564273</th>\n",
       "    <th>1.886523</th>\n",
       "    <th>0.340700</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>37</th>\n",
       "    <th>1.540024</th>\n",
       "    <th>1.830607</th>\n",
       "    <th>0.348000</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>38</th>\n",
       "    <th>1.523219</th>\n",
       "    <th>1.829362</th>\n",
       "    <th>0.357700</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>39</th>\n",
       "    <th>1.498133</th>\n",
       "    <th>1.830962</th>\n",
       "    <th>0.349800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>40</th>\n",
       "    <th>1.480417</th>\n",
       "    <th>1.787680</th>\n",
       "    <th>0.356200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>41</th>\n",
       "    <th>1.465634</th>\n",
       "    <th>1.777595</th>\n",
       "    <th>0.358900</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>42</th>\n",
       "    <th>1.452270</th>\n",
       "    <th>1.785595</th>\n",
       "    <th>0.357400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>43</th>\n",
       "    <th>1.438567</th>\n",
       "    <th>1.761703</th>\n",
       "    <th>0.363500</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>44</th>\n",
       "    <th>1.421039</th>\n",
       "    <th>1.759870</th>\n",
       "    <th>0.369200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>45</th>\n",
       "    <th>1.408826</th>\n",
       "    <th>1.749598</th>\n",
       "    <th>0.370400</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>46</th>\n",
       "    <th>1.393510</th>\n",
       "    <th>1.747831</th>\n",
       "    <th>0.372200</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>47</th>\n",
       "    <th>1.376800</th>\n",
       "    <th>1.748050</th>\n",
       "    <th>0.372000</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>48</th>\n",
       "    <th>1.362877</th>\n",
       "    <th>1.747384</th>\n",
       "    <th>0.370800</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>49</th>\n",
       "    <th>1.345895</th>\n",
       "    <th>1.742733</th>\n",
       "    <th>0.372300</th>\n",
       "    <th>00:08</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learnBase = Learner(data_labeled,models.WideResNet(num_groups=3,N=4,num_classes=10,k=2,start_nf=32),metrics=accuracy)\n",
    "learnBase.fit_one_cycle(50,2e-2,wd=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally I will train with MixMatch, using the learner defined in the previous section. Note that I now use `learn.fit` instead of `learn.fit_one_cycle` since MixMatch uses EMA instead of a learning rate schedule. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='187' class='' max='200', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      93.50% [187/200 14:07:39<58:55]\n",
       "    </div>\n",
       "    \n",
       "<table style='width:525px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>l_loss</th>\n",
       "    <th>ul_loss</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>1.768446</th>\n",
       "    <th>1.692881</th>\n",
       "    <th>0.389600</th>\n",
       "    <th>1.677018</th>\n",
       "    <th>0.008062</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>1.623252</th>\n",
       "    <th>1.635129</th>\n",
       "    <th>0.432200</th>\n",
       "    <th>1.342663</th>\n",
       "    <th>0.011633</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>1.591076</th>\n",
       "    <th>1.659631</th>\n",
       "    <th>0.442200</th>\n",
       "    <th>1.164911</th>\n",
       "    <th>0.011560</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>1.707859</th>\n",
       "    <th>1.626472</th>\n",
       "    <th>0.472800</th>\n",
       "    <th>1.100260</th>\n",
       "    <th>0.012204</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>1.748360</th>\n",
       "    <th>1.692946</th>\n",
       "    <th>0.469300</th>\n",
       "    <th>1.037391</th>\n",
       "    <th>0.011352</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.981373</th>\n",
       "    <th>1.682426</th>\n",
       "    <th>0.481600</th>\n",
       "    <th>1.089962</th>\n",
       "    <th>0.011801</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>2.238199</th>\n",
       "    <th>1.659572</th>\n",
       "    <th>0.494100</th>\n",
       "    <th>1.103848</th>\n",
       "    <th>0.012825</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>2.401606</th>\n",
       "    <th>1.676859</th>\n",
       "    <th>0.486100</th>\n",
       "    <th>1.113788</th>\n",
       "    <th>0.012912</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>2.275591</th>\n",
       "    <th>1.773682</th>\n",
       "    <th>0.486400</th>\n",
       "    <th>1.061513</th>\n",
       "    <th>0.012141</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>2.122196</th>\n",
       "    <th>1.780571</th>\n",
       "    <th>0.522700</th>\n",
       "    <th>0.980808</th>\n",
       "    <th>0.011414</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>2.112240</th>\n",
       "    <th>1.629649</th>\n",
       "    <th>0.546900</th>\n",
       "    <th>0.963559</th>\n",
       "    <th>0.011487</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>2.309766</th>\n",
       "    <th>1.598194</th>\n",
       "    <th>0.540000</th>\n",
       "    <th>1.037701</th>\n",
       "    <th>0.012721</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>2.194480</th>\n",
       "    <th>1.596004</th>\n",
       "    <th>0.541700</th>\n",
       "    <th>1.000109</th>\n",
       "    <th>0.011944</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>2.345027</th>\n",
       "    <th>1.395312</th>\n",
       "    <th>0.582600</th>\n",
       "    <th>1.070695</th>\n",
       "    <th>0.012743</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>2.270950</th>\n",
       "    <th>1.689884</th>\n",
       "    <th>0.531400</th>\n",
       "    <th>1.012973</th>\n",
       "    <th>0.012580</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>2.129356</th>\n",
       "    <th>1.434179</th>\n",
       "    <th>0.585600</th>\n",
       "    <th>0.986743</th>\n",
       "    <th>0.011426</th>\n",
       "    <th>04:24</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>2.157623</th>\n",
       "    <th>1.253479</th>\n",
       "    <th>0.627500</th>\n",
       "    <th>0.975062</th>\n",
       "    <th>0.011826</th>\n",
       "    <th>04:24</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>2.089340</th>\n",
       "    <th>1.386314</th>\n",
       "    <th>0.598100</th>\n",
       "    <th>0.933850</th>\n",
       "    <th>0.011555</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>2.207202</th>\n",
       "    <th>1.316900</th>\n",
       "    <th>0.612100</th>\n",
       "    <th>0.997327</th>\n",
       "    <th>0.012099</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>1.911509</th>\n",
       "    <th>1.342133</th>\n",
       "    <th>0.639400</th>\n",
       "    <th>0.866124</th>\n",
       "    <th>0.010454</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>2.071062</th>\n",
       "    <th>1.147844</th>\n",
       "    <th>0.670600</th>\n",
       "    <th>0.952565</th>\n",
       "    <th>0.011185</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>21</th>\n",
       "    <th>2.083739</th>\n",
       "    <th>1.189340</th>\n",
       "    <th>0.661400</th>\n",
       "    <th>0.933829</th>\n",
       "    <th>0.011499</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>22</th>\n",
       "    <th>2.140740</th>\n",
       "    <th>1.164990</th>\n",
       "    <th>0.672800</th>\n",
       "    <th>0.965642</th>\n",
       "    <th>0.011751</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>23</th>\n",
       "    <th>1.995096</th>\n",
       "    <th>1.140034</th>\n",
       "    <th>0.687300</th>\n",
       "    <th>0.896218</th>\n",
       "    <th>0.010989</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>24</th>\n",
       "    <th>2.206992</th>\n",
       "    <th>1.064863</th>\n",
       "    <th>0.690000</th>\n",
       "    <th>0.995957</th>\n",
       "    <th>0.012110</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>25</th>\n",
       "    <th>2.064593</th>\n",
       "    <th>1.119878</th>\n",
       "    <th>0.689000</th>\n",
       "    <th>0.925864</th>\n",
       "    <th>0.011387</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>26</th>\n",
       "    <th>1.961534</th>\n",
       "    <th>0.944794</th>\n",
       "    <th>0.734000</th>\n",
       "    <th>0.880994</th>\n",
       "    <th>0.010805</th>\n",
       "    <th>04:25</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>27</th>\n",
       "    <th>2.068559</th>\n",
       "    <th>1.031900</th>\n",
       "    <th>0.707000</th>\n",
       "    <th>0.925561</th>\n",
       "    <th>0.011430</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>28</th>\n",
       "    <th>1.872903</th>\n",
       "    <th>1.137156</th>\n",
       "    <th>0.703300</th>\n",
       "    <th>0.819757</th>\n",
       "    <th>0.010531</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>29</th>\n",
       "    <th>2.127771</th>\n",
       "    <th>0.952049</th>\n",
       "    <th>0.720500</th>\n",
       "    <th>0.947353</th>\n",
       "    <th>0.011804</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>30</th>\n",
       "    <th>1.928028</th>\n",
       "    <th>0.994461</th>\n",
       "    <th>0.722500</th>\n",
       "    <th>0.868828</th>\n",
       "    <th>0.010592</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>31</th>\n",
       "    <th>1.954530</th>\n",
       "    <th>1.108844</th>\n",
       "    <th>0.709000</th>\n",
       "    <th>0.891667</th>\n",
       "    <th>0.010629</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>32</th>\n",
       "    <th>1.844156</th>\n",
       "    <th>0.985746</th>\n",
       "    <th>0.724800</th>\n",
       "    <th>0.834487</th>\n",
       "    <th>0.010097</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>33</th>\n",
       "    <th>2.030218</th>\n",
       "    <th>0.954471</th>\n",
       "    <th>0.733200</th>\n",
       "    <th>0.892671</th>\n",
       "    <th>0.011375</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>34</th>\n",
       "    <th>1.923161</th>\n",
       "    <th>0.942925</th>\n",
       "    <th>0.742300</th>\n",
       "    <th>0.864076</th>\n",
       "    <th>0.010591</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>35</th>\n",
       "    <th>1.837016</th>\n",
       "    <th>0.983558</th>\n",
       "    <th>0.730500</th>\n",
       "    <th>0.823773</th>\n",
       "    <th>0.010132</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>36</th>\n",
       "    <th>1.897623</th>\n",
       "    <th>0.930361</th>\n",
       "    <th>0.746300</th>\n",
       "    <th>0.863888</th>\n",
       "    <th>0.010337</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>37</th>\n",
       "    <th>2.102383</th>\n",
       "    <th>0.931452</th>\n",
       "    <th>0.746900</th>\n",
       "    <th>0.934087</th>\n",
       "    <th>0.011683</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>38</th>\n",
       "    <th>1.807780</th>\n",
       "    <th>0.967259</th>\n",
       "    <th>0.747500</th>\n",
       "    <th>0.816519</th>\n",
       "    <th>0.009913</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>39</th>\n",
       "    <th>1.938055</th>\n",
       "    <th>0.814318</th>\n",
       "    <th>0.781100</th>\n",
       "    <th>0.874709</th>\n",
       "    <th>0.010633</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>40</th>\n",
       "    <th>1.960714</th>\n",
       "    <th>0.817678</th>\n",
       "    <th>0.772600</th>\n",
       "    <th>0.880872</th>\n",
       "    <th>0.010798</th>\n",
       "    <th>04:36</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>41</th>\n",
       "    <th>2.047494</th>\n",
       "    <th>0.798482</th>\n",
       "    <th>0.780800</th>\n",
       "    <th>0.914399</th>\n",
       "    <th>0.011331</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>42</th>\n",
       "    <th>1.879855</th>\n",
       "    <th>0.917445</th>\n",
       "    <th>0.747600</th>\n",
       "    <th>0.837697</th>\n",
       "    <th>0.010422</th>\n",
       "    <th>04:35</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>43</th>\n",
       "    <th>1.812956</th>\n",
       "    <th>0.870310</th>\n",
       "    <th>0.765300</th>\n",
       "    <th>0.823754</th>\n",
       "    <th>0.009892</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>44</th>\n",
       "    <th>1.951769</th>\n",
       "    <th>0.873486</th>\n",
       "    <th>0.758800</th>\n",
       "    <th>0.886743</th>\n",
       "    <th>0.010650</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>45</th>\n",
       "    <th>1.956069</th>\n",
       "    <th>0.839884</th>\n",
       "    <th>0.760700</th>\n",
       "    <th>0.881612</th>\n",
       "    <th>0.010745</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>46</th>\n",
       "    <th>1.831361</th>\n",
       "    <th>0.844626</th>\n",
       "    <th>0.774600</th>\n",
       "    <th>0.842265</th>\n",
       "    <th>0.009891</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>47</th>\n",
       "    <th>1.757242</th>\n",
       "    <th>0.816120</th>\n",
       "    <th>0.783100</th>\n",
       "    <th>0.789299</th>\n",
       "    <th>0.009679</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>48</th>\n",
       "    <th>1.923197</th>\n",
       "    <th>0.795811</th>\n",
       "    <th>0.788300</th>\n",
       "    <th>0.864741</th>\n",
       "    <th>0.010585</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>49</th>\n",
       "    <th>1.851986</th>\n",
       "    <th>0.772088</th>\n",
       "    <th>0.785900</th>\n",
       "    <th>0.839377</th>\n",
       "    <th>0.010126</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>50</th>\n",
       "    <th>1.876624</th>\n",
       "    <th>0.795911</th>\n",
       "    <th>0.778200</th>\n",
       "    <th>0.837684</th>\n",
       "    <th>0.010389</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>51</th>\n",
       "    <th>1.822752</th>\n",
       "    <th>0.804463</th>\n",
       "    <th>0.783200</th>\n",
       "    <th>0.828995</th>\n",
       "    <th>0.009938</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>52</th>\n",
       "    <th>1.866998</th>\n",
       "    <th>0.825565</th>\n",
       "    <th>0.775700</th>\n",
       "    <th>0.851229</th>\n",
       "    <th>0.010158</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>53</th>\n",
       "    <th>1.897805</th>\n",
       "    <th>0.750035</th>\n",
       "    <th>0.800900</th>\n",
       "    <th>0.845294</th>\n",
       "    <th>0.010525</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>54</th>\n",
       "    <th>1.651654</th>\n",
       "    <th>0.833203</th>\n",
       "    <th>0.780900</th>\n",
       "    <th>0.749399</th>\n",
       "    <th>0.009023</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>55</th>\n",
       "    <th>1.848239</th>\n",
       "    <th>0.811754</th>\n",
       "    <th>0.789100</th>\n",
       "    <th>0.828780</th>\n",
       "    <th>0.010195</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>56</th>\n",
       "    <th>1.843379</th>\n",
       "    <th>0.838132</th>\n",
       "    <th>0.777400</th>\n",
       "    <th>0.833663</th>\n",
       "    <th>0.010097</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>57</th>\n",
       "    <th>1.778225</th>\n",
       "    <th>0.803397</th>\n",
       "    <th>0.783800</th>\n",
       "    <th>0.815787</th>\n",
       "    <th>0.009624</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>58</th>\n",
       "    <th>1.912916</th>\n",
       "    <th>0.786136</th>\n",
       "    <th>0.795300</th>\n",
       "    <th>0.857526</th>\n",
       "    <th>0.010554</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>59</th>\n",
       "    <th>1.682768</th>\n",
       "    <th>0.723798</th>\n",
       "    <th>0.806600</th>\n",
       "    <th>0.784702</th>\n",
       "    <th>0.008981</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>60</th>\n",
       "    <th>1.809915</th>\n",
       "    <th>0.744520</th>\n",
       "    <th>0.802700</th>\n",
       "    <th>0.807832</th>\n",
       "    <th>0.010021</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>61</th>\n",
       "    <th>1.765583</th>\n",
       "    <th>0.786174</th>\n",
       "    <th>0.787100</th>\n",
       "    <th>0.792369</th>\n",
       "    <th>0.009732</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>62</th>\n",
       "    <th>1.768791</th>\n",
       "    <th>0.682424</th>\n",
       "    <th>0.814300</th>\n",
       "    <th>0.807605</th>\n",
       "    <th>0.009612</th>\n",
       "    <th>04:25</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>63</th>\n",
       "    <th>1.773594</th>\n",
       "    <th>0.790723</th>\n",
       "    <th>0.796200</th>\n",
       "    <th>0.800271</th>\n",
       "    <th>0.009733</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>64</th>\n",
       "    <th>1.856683</th>\n",
       "    <th>0.679510</th>\n",
       "    <th>0.815500</th>\n",
       "    <th>0.840930</th>\n",
       "    <th>0.010158</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>65</th>\n",
       "    <th>1.791854</th>\n",
       "    <th>0.697147</th>\n",
       "    <th>0.811500</th>\n",
       "    <th>0.808415</th>\n",
       "    <th>0.009834</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>66</th>\n",
       "    <th>1.552317</th>\n",
       "    <th>0.707006</th>\n",
       "    <th>0.816200</th>\n",
       "    <th>0.713796</th>\n",
       "    <th>0.008385</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>67</th>\n",
       "    <th>1.649572</th>\n",
       "    <th>0.751610</th>\n",
       "    <th>0.802400</th>\n",
       "    <th>0.755373</th>\n",
       "    <th>0.008942</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>68</th>\n",
       "    <th>1.718790</th>\n",
       "    <th>0.701873</th>\n",
       "    <th>0.810900</th>\n",
       "    <th>0.778481</th>\n",
       "    <th>0.009403</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>69</th>\n",
       "    <th>1.565507</th>\n",
       "    <th>0.776781</th>\n",
       "    <th>0.800000</th>\n",
       "    <th>0.727624</th>\n",
       "    <th>0.008379</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>70</th>\n",
       "    <th>1.733020</th>\n",
       "    <th>0.693333</th>\n",
       "    <th>0.818400</th>\n",
       "    <th>0.790340</th>\n",
       "    <th>0.009427</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>71</th>\n",
       "    <th>1.586215</th>\n",
       "    <th>0.802036</th>\n",
       "    <th>0.796700</th>\n",
       "    <th>0.736719</th>\n",
       "    <th>0.008495</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>72</th>\n",
       "    <th>1.714591</th>\n",
       "    <th>0.734201</th>\n",
       "    <th>0.814100</th>\n",
       "    <th>0.796071</th>\n",
       "    <th>0.009185</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>73</th>\n",
       "    <th>1.688499</th>\n",
       "    <th>0.645813</th>\n",
       "    <th>0.834900</th>\n",
       "    <th>0.778959</th>\n",
       "    <th>0.009095</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>74</th>\n",
       "    <th>1.724636</th>\n",
       "    <th>0.647961</th>\n",
       "    <th>0.829600</th>\n",
       "    <th>0.800833</th>\n",
       "    <th>0.009238</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>75</th>\n",
       "    <th>1.730815</th>\n",
       "    <th>0.638836</th>\n",
       "    <th>0.833100</th>\n",
       "    <th>0.793784</th>\n",
       "    <th>0.009370</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>76</th>\n",
       "    <th>1.649783</th>\n",
       "    <th>0.620173</th>\n",
       "    <th>0.834800</th>\n",
       "    <th>0.759069</th>\n",
       "    <th>0.008907</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>77</th>\n",
       "    <th>1.677960</th>\n",
       "    <th>0.694792</th>\n",
       "    <th>0.813000</th>\n",
       "    <th>0.764615</th>\n",
       "    <th>0.009133</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>78</th>\n",
       "    <th>1.589998</th>\n",
       "    <th>0.749398</th>\n",
       "    <th>0.807200</th>\n",
       "    <th>0.728117</th>\n",
       "    <th>0.008619</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>79</th>\n",
       "    <th>1.680000</th>\n",
       "    <th>0.637823</th>\n",
       "    <th>0.837400</th>\n",
       "    <th>0.788180</th>\n",
       "    <th>0.008918</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>80</th>\n",
       "    <th>1.652816</th>\n",
       "    <th>0.660567</th>\n",
       "    <th>0.827200</th>\n",
       "    <th>0.759389</th>\n",
       "    <th>0.008934</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>81</th>\n",
       "    <th>1.742141</th>\n",
       "    <th>0.651616</th>\n",
       "    <th>0.827000</th>\n",
       "    <th>0.805965</th>\n",
       "    <th>0.009362</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>82</th>\n",
       "    <th>1.651093</th>\n",
       "    <th>0.634614</th>\n",
       "    <th>0.835600</th>\n",
       "    <th>0.737018</th>\n",
       "    <th>0.009141</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>83</th>\n",
       "    <th>1.645259</th>\n",
       "    <th>0.680019</th>\n",
       "    <th>0.827700</th>\n",
       "    <th>0.762275</th>\n",
       "    <th>0.008830</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>84</th>\n",
       "    <th>1.609792</th>\n",
       "    <th>0.774589</th>\n",
       "    <th>0.804800</th>\n",
       "    <th>0.748437</th>\n",
       "    <th>0.008614</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>85</th>\n",
       "    <th>1.606779</th>\n",
       "    <th>0.644867</th>\n",
       "    <th>0.833900</th>\n",
       "    <th>0.753266</th>\n",
       "    <th>0.008535</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>86</th>\n",
       "    <th>1.543270</th>\n",
       "    <th>0.681161</th>\n",
       "    <th>0.829500</th>\n",
       "    <th>0.718942</th>\n",
       "    <th>0.008243</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>87</th>\n",
       "    <th>1.622073</th>\n",
       "    <th>0.579168</th>\n",
       "    <th>0.853200</th>\n",
       "    <th>0.762207</th>\n",
       "    <th>0.008599</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>88</th>\n",
       "    <th>1.679510</th>\n",
       "    <th>0.677916</th>\n",
       "    <th>0.833500</th>\n",
       "    <th>0.778213</th>\n",
       "    <th>0.009013</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>89</th>\n",
       "    <th>1.541078</th>\n",
       "    <th>0.644096</th>\n",
       "    <th>0.838600</th>\n",
       "    <th>0.720482</th>\n",
       "    <th>0.008206</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>90</th>\n",
       "    <th>1.496218</th>\n",
       "    <th>0.675837</th>\n",
       "    <th>0.831700</th>\n",
       "    <th>0.706531</th>\n",
       "    <th>0.007897</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>91</th>\n",
       "    <th>1.487647</th>\n",
       "    <th>0.617808</th>\n",
       "    <th>0.846100</th>\n",
       "    <th>0.699701</th>\n",
       "    <th>0.007879</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>92</th>\n",
       "    <th>1.611398</th>\n",
       "    <th>0.667796</th>\n",
       "    <th>0.829400</th>\n",
       "    <th>0.757409</th>\n",
       "    <th>0.008540</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>93</th>\n",
       "    <th>1.643473</th>\n",
       "    <th>0.612963</th>\n",
       "    <th>0.841000</th>\n",
       "    <th>0.767751</th>\n",
       "    <th>0.008757</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>94</th>\n",
       "    <th>1.416742</th>\n",
       "    <th>0.639008</th>\n",
       "    <th>0.837500</th>\n",
       "    <th>0.662027</th>\n",
       "    <th>0.007547</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>95</th>\n",
       "    <th>1.573668</th>\n",
       "    <th>0.651981</th>\n",
       "    <th>0.835400</th>\n",
       "    <th>0.728186</th>\n",
       "    <th>0.008455</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>96</th>\n",
       "    <th>1.438987</th>\n",
       "    <th>0.628358</th>\n",
       "    <th>0.845000</th>\n",
       "    <th>0.676523</th>\n",
       "    <th>0.007625</th>\n",
       "    <th>04:26</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>97</th>\n",
       "    <th>1.442265</th>\n",
       "    <th>0.652476</th>\n",
       "    <th>0.841600</th>\n",
       "    <th>0.686659</th>\n",
       "    <th>0.007556</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>98</th>\n",
       "    <th>1.477257</th>\n",
       "    <th>0.642103</th>\n",
       "    <th>0.836200</th>\n",
       "    <th>0.700708</th>\n",
       "    <th>0.007765</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>99</th>\n",
       "    <th>1.590611</th>\n",
       "    <th>0.655946</th>\n",
       "    <th>0.829800</th>\n",
       "    <th>0.763361</th>\n",
       "    <th>0.008272</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>100</th>\n",
       "    <th>1.554803</th>\n",
       "    <th>0.630596</th>\n",
       "    <th>0.845000</th>\n",
       "    <th>0.733557</th>\n",
       "    <th>0.008212</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>101</th>\n",
       "    <th>1.588827</th>\n",
       "    <th>0.686192</th>\n",
       "    <th>0.832500</th>\n",
       "    <th>0.753247</th>\n",
       "    <th>0.008356</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>102</th>\n",
       "    <th>1.390303</th>\n",
       "    <th>0.644003</th>\n",
       "    <th>0.839100</th>\n",
       "    <th>0.674573</th>\n",
       "    <th>0.007157</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>103</th>\n",
       "    <th>1.598292</th>\n",
       "    <th>0.608365</th>\n",
       "    <th>0.847400</th>\n",
       "    <th>0.760304</th>\n",
       "    <th>0.008380</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>104</th>\n",
       "    <th>1.407854</th>\n",
       "    <th>0.599136</th>\n",
       "    <th>0.853100</th>\n",
       "    <th>0.689678</th>\n",
       "    <th>0.007182</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>105</th>\n",
       "    <th>1.548669</th>\n",
       "    <th>0.613295</th>\n",
       "    <th>0.842000</th>\n",
       "    <th>0.723408</th>\n",
       "    <th>0.008253</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>106</th>\n",
       "    <th>1.488103</th>\n",
       "    <th>0.662287</th>\n",
       "    <th>0.832100</th>\n",
       "    <th>0.720261</th>\n",
       "    <th>0.007678</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>107</th>\n",
       "    <th>1.505939</th>\n",
       "    <th>0.587630</th>\n",
       "    <th>0.856600</th>\n",
       "    <th>0.711261</th>\n",
       "    <th>0.007947</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>108</th>\n",
       "    <th>1.500859</th>\n",
       "    <th>0.609804</th>\n",
       "    <th>0.846600</th>\n",
       "    <th>0.706366</th>\n",
       "    <th>0.007945</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>109</th>\n",
       "    <th>1.523265</th>\n",
       "    <th>0.612768</th>\n",
       "    <th>0.849100</th>\n",
       "    <th>0.720826</th>\n",
       "    <th>0.008024</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>110</th>\n",
       "    <th>1.470100</th>\n",
       "    <th>0.588600</th>\n",
       "    <th>0.852200</th>\n",
       "    <th>0.704469</th>\n",
       "    <th>0.007656</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>111</th>\n",
       "    <th>1.487828</th>\n",
       "    <th>0.597848</th>\n",
       "    <th>0.853700</th>\n",
       "    <th>0.710621</th>\n",
       "    <th>0.007772</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>112</th>\n",
       "    <th>1.312327</th>\n",
       "    <th>0.579094</th>\n",
       "    <th>0.861100</th>\n",
       "    <th>0.636656</th>\n",
       "    <th>0.006757</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>113</th>\n",
       "    <th>1.572592</th>\n",
       "    <th>0.585187</th>\n",
       "    <th>0.857100</th>\n",
       "    <th>0.753036</th>\n",
       "    <th>0.008196</th>\n",
       "    <th>04:25</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>114</th>\n",
       "    <th>1.588859</th>\n",
       "    <th>0.646963</th>\n",
       "    <th>0.842700</th>\n",
       "    <th>0.735416</th>\n",
       "    <th>0.008534</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>115</th>\n",
       "    <th>1.265692</th>\n",
       "    <th>0.573558</th>\n",
       "    <th>0.856100</th>\n",
       "    <th>0.608627</th>\n",
       "    <th>0.006571</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>116</th>\n",
       "    <th>1.432645</th>\n",
       "    <th>0.545162</th>\n",
       "    <th>0.866600</th>\n",
       "    <th>0.694596</th>\n",
       "    <th>0.007380</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>117</th>\n",
       "    <th>1.328743</th>\n",
       "    <th>0.560207</th>\n",
       "    <th>0.862200</th>\n",
       "    <th>0.663284</th>\n",
       "    <th>0.006655</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>118</th>\n",
       "    <th>1.522755</th>\n",
       "    <th>0.545430</th>\n",
       "    <th>0.865600</th>\n",
       "    <th>0.720001</th>\n",
       "    <th>0.008028</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>119</th>\n",
       "    <th>1.438829</th>\n",
       "    <th>0.605778</th>\n",
       "    <th>0.854800</th>\n",
       "    <th>0.700525</th>\n",
       "    <th>0.007383</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>120</th>\n",
       "    <th>1.414728</th>\n",
       "    <th>0.560969</th>\n",
       "    <th>0.861900</th>\n",
       "    <th>0.685853</th>\n",
       "    <th>0.007289</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>121</th>\n",
       "    <th>1.387415</th>\n",
       "    <th>0.542830</th>\n",
       "    <th>0.868700</th>\n",
       "    <th>0.674739</th>\n",
       "    <th>0.007127</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>122</th>\n",
       "    <th>1.599676</th>\n",
       "    <th>0.552790</th>\n",
       "    <th>0.861900</th>\n",
       "    <th>0.757339</th>\n",
       "    <th>0.008423</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>123</th>\n",
       "    <th>1.307125</th>\n",
       "    <th>0.563348</th>\n",
       "    <th>0.860700</th>\n",
       "    <th>0.654039</th>\n",
       "    <th>0.006531</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>124</th>\n",
       "    <th>1.471041</th>\n",
       "    <th>0.585730</th>\n",
       "    <th>0.856700</th>\n",
       "    <th>0.728879</th>\n",
       "    <th>0.007422</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>125</th>\n",
       "    <th>1.400870</th>\n",
       "    <th>0.555772</th>\n",
       "    <th>0.867200</th>\n",
       "    <th>0.680815</th>\n",
       "    <th>0.007201</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>126</th>\n",
       "    <th>1.456615</th>\n",
       "    <th>0.568775</th>\n",
       "    <th>0.861600</th>\n",
       "    <th>0.687363</th>\n",
       "    <th>0.007693</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>127</th>\n",
       "    <th>1.355280</th>\n",
       "    <th>0.554532</th>\n",
       "    <th>0.862300</th>\n",
       "    <th>0.676087</th>\n",
       "    <th>0.006792</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>128</th>\n",
       "    <th>1.449984</th>\n",
       "    <th>0.561475</th>\n",
       "    <th>0.861100</th>\n",
       "    <th>0.714712</th>\n",
       "    <th>0.007353</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>129</th>\n",
       "    <th>1.367693</th>\n",
       "    <th>0.574884</th>\n",
       "    <th>0.864700</th>\n",
       "    <th>0.669527</th>\n",
       "    <th>0.006982</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>130</th>\n",
       "    <th>1.475356</th>\n",
       "    <th>0.530124</th>\n",
       "    <th>0.870800</th>\n",
       "    <th>0.709647</th>\n",
       "    <th>0.007657</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>131</th>\n",
       "    <th>1.374049</th>\n",
       "    <th>0.556036</th>\n",
       "    <th>0.865600</th>\n",
       "    <th>0.693531</th>\n",
       "    <th>0.006805</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>132</th>\n",
       "    <th>1.285313</th>\n",
       "    <th>0.610216</th>\n",
       "    <th>0.858400</th>\n",
       "    <th>0.644021</th>\n",
       "    <th>0.006413</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>133</th>\n",
       "    <th>1.311746</th>\n",
       "    <th>0.573795</th>\n",
       "    <th>0.863100</th>\n",
       "    <th>0.651910</th>\n",
       "    <th>0.006598</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>134</th>\n",
       "    <th>1.402064</th>\n",
       "    <th>0.551591</th>\n",
       "    <th>0.869400</th>\n",
       "    <th>0.695125</th>\n",
       "    <th>0.007069</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>135</th>\n",
       "    <th>1.368074</th>\n",
       "    <th>0.545775</th>\n",
       "    <th>0.869900</th>\n",
       "    <th>0.684859</th>\n",
       "    <th>0.006832</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>136</th>\n",
       "    <th>1.414046</th>\n",
       "    <th>0.557015</th>\n",
       "    <th>0.865300</th>\n",
       "    <th>0.704621</th>\n",
       "    <th>0.007094</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>137</th>\n",
       "    <th>1.467728</th>\n",
       "    <th>0.521244</th>\n",
       "    <th>0.874000</th>\n",
       "    <th>0.717204</th>\n",
       "    <th>0.007505</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>138</th>\n",
       "    <th>1.293745</th>\n",
       "    <th>0.547094</th>\n",
       "    <th>0.869600</th>\n",
       "    <th>0.662614</th>\n",
       "    <th>0.006311</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>139</th>\n",
       "    <th>1.467425</th>\n",
       "    <th>0.549886</th>\n",
       "    <th>0.870500</th>\n",
       "    <th>0.729408</th>\n",
       "    <th>0.007380</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>140</th>\n",
       "    <th>1.487326</th>\n",
       "    <th>0.515896</th>\n",
       "    <th>0.876700</th>\n",
       "    <th>0.746436</th>\n",
       "    <th>0.007409</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>141</th>\n",
       "    <th>1.333704</th>\n",
       "    <th>0.559527</th>\n",
       "    <th>0.867500</th>\n",
       "    <th>0.679032</th>\n",
       "    <th>0.006547</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>142</th>\n",
       "    <th>1.258717</th>\n",
       "    <th>0.539584</th>\n",
       "    <th>0.871200</th>\n",
       "    <th>0.641496</th>\n",
       "    <th>0.006172</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>143</th>\n",
       "    <th>1.323165</th>\n",
       "    <th>0.552072</th>\n",
       "    <th>0.868200</th>\n",
       "    <th>0.682777</th>\n",
       "    <th>0.006404</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>144</th>\n",
       "    <th>1.319582</th>\n",
       "    <th>0.539186</th>\n",
       "    <th>0.874400</th>\n",
       "    <th>0.673250</th>\n",
       "    <th>0.006463</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>145</th>\n",
       "    <th>1.385945</th>\n",
       "    <th>0.554222</th>\n",
       "    <th>0.871100</th>\n",
       "    <th>0.688976</th>\n",
       "    <th>0.006970</th>\n",
       "    <th>04:34</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>146</th>\n",
       "    <th>1.330253</th>\n",
       "    <th>0.535862</th>\n",
       "    <th>0.873400</th>\n",
       "    <th>0.682822</th>\n",
       "    <th>0.006474</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>147</th>\n",
       "    <th>1.349451</th>\n",
       "    <th>0.540253</th>\n",
       "    <th>0.873100</th>\n",
       "    <th>0.675145</th>\n",
       "    <th>0.006743</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>148</th>\n",
       "    <th>1.341914</th>\n",
       "    <th>0.539603</th>\n",
       "    <th>0.875100</th>\n",
       "    <th>0.689932</th>\n",
       "    <th>0.006520</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>149</th>\n",
       "    <th>1.228144</th>\n",
       "    <th>0.536138</th>\n",
       "    <th>0.874300</th>\n",
       "    <th>0.621357</th>\n",
       "    <th>0.006068</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>150</th>\n",
       "    <th>1.356122</th>\n",
       "    <th>0.540268</th>\n",
       "    <th>0.872100</th>\n",
       "    <th>0.684577</th>\n",
       "    <th>0.006715</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>151</th>\n",
       "    <th>1.346776</th>\n",
       "    <th>0.548479</th>\n",
       "    <th>0.872000</th>\n",
       "    <th>0.661993</th>\n",
       "    <th>0.006848</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>152</th>\n",
       "    <th>1.251722</th>\n",
       "    <th>0.568449</th>\n",
       "    <th>0.866700</th>\n",
       "    <th>0.647495</th>\n",
       "    <th>0.006042</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>153</th>\n",
       "    <th>1.387145</th>\n",
       "    <th>0.523135</th>\n",
       "    <th>0.876700</th>\n",
       "    <th>0.694046</th>\n",
       "    <th>0.006931</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>154</th>\n",
       "    <th>1.274961</th>\n",
       "    <th>0.552718</th>\n",
       "    <th>0.871000</th>\n",
       "    <th>0.649362</th>\n",
       "    <th>0.006256</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>155</th>\n",
       "    <th>1.433880</th>\n",
       "    <th>0.529421</th>\n",
       "    <th>0.875700</th>\n",
       "    <th>0.718615</th>\n",
       "    <th>0.007153</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>156</th>\n",
       "    <th>1.221045</th>\n",
       "    <th>0.536909</th>\n",
       "    <th>0.871600</th>\n",
       "    <th>0.648637</th>\n",
       "    <th>0.005724</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>157</th>\n",
       "    <th>1.239568</th>\n",
       "    <th>0.565668</th>\n",
       "    <th>0.870700</th>\n",
       "    <th>0.629598</th>\n",
       "    <th>0.006100</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>158</th>\n",
       "    <th>1.202694</th>\n",
       "    <th>0.533253</th>\n",
       "    <th>0.876800</th>\n",
       "    <th>0.621021</th>\n",
       "    <th>0.005817</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>159</th>\n",
       "    <th>1.262689</th>\n",
       "    <th>0.550639</th>\n",
       "    <th>0.875900</th>\n",
       "    <th>0.652829</th>\n",
       "    <th>0.006099</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>160</th>\n",
       "    <th>1.304296</th>\n",
       "    <th>0.537292</th>\n",
       "    <th>0.876800</th>\n",
       "    <th>0.666835</th>\n",
       "    <th>0.006375</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>161</th>\n",
       "    <th>1.105957</th>\n",
       "    <th>0.535096</th>\n",
       "    <th>0.875000</th>\n",
       "    <th>0.582732</th>\n",
       "    <th>0.005232</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>162</th>\n",
       "    <th>1.205164</th>\n",
       "    <th>0.547447</th>\n",
       "    <th>0.875100</th>\n",
       "    <th>0.649703</th>\n",
       "    <th>0.005555</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>163</th>\n",
       "    <th>1.376448</th>\n",
       "    <th>0.525952</th>\n",
       "    <th>0.876900</th>\n",
       "    <th>0.706548</th>\n",
       "    <th>0.006699</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>164</th>\n",
       "    <th>1.308145</th>\n",
       "    <th>0.535083</th>\n",
       "    <th>0.875600</th>\n",
       "    <th>0.676386</th>\n",
       "    <th>0.006318</th>\n",
       "    <th>04:27</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>165</th>\n",
       "    <th>1.357478</th>\n",
       "    <th>0.538852</th>\n",
       "    <th>0.876600</th>\n",
       "    <th>0.694311</th>\n",
       "    <th>0.006632</th>\n",
       "    <th>04:29</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>166</th>\n",
       "    <th>1.145550</th>\n",
       "    <th>0.544624</th>\n",
       "    <th>0.877600</th>\n",
       "    <th>0.602523</th>\n",
       "    <th>0.005430</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>167</th>\n",
       "    <th>1.268494</th>\n",
       "    <th>0.544500</th>\n",
       "    <th>0.872300</th>\n",
       "    <th>0.673103</th>\n",
       "    <th>0.005954</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>168</th>\n",
       "    <th>1.146509</th>\n",
       "    <th>0.526813</th>\n",
       "    <th>0.876800</th>\n",
       "    <th>0.616976</th>\n",
       "    <th>0.005295</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>169</th>\n",
       "    <th>1.206311</th>\n",
       "    <th>0.556149</th>\n",
       "    <th>0.876400</th>\n",
       "    <th>0.637541</th>\n",
       "    <th>0.005688</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>170</th>\n",
       "    <th>1.108583</th>\n",
       "    <th>0.557292</th>\n",
       "    <th>0.876500</th>\n",
       "    <th>0.587324</th>\n",
       "    <th>0.005213</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>171</th>\n",
       "    <th>1.273828</th>\n",
       "    <th>0.526203</th>\n",
       "    <th>0.876600</th>\n",
       "    <th>0.655114</th>\n",
       "    <th>0.006187</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>172</th>\n",
       "    <th>1.257936</th>\n",
       "    <th>0.530462</th>\n",
       "    <th>0.878500</th>\n",
       "    <th>0.668464</th>\n",
       "    <th>0.005895</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>173</th>\n",
       "    <th>1.277861</th>\n",
       "    <th>0.523026</th>\n",
       "    <th>0.882000</th>\n",
       "    <th>0.653293</th>\n",
       "    <th>0.006246</th>\n",
       "    <th>04:28</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>174</th>\n",
       "    <th>1.161005</th>\n",
       "    <th>0.554460</th>\n",
       "    <th>0.876600</th>\n",
       "    <th>0.629368</th>\n",
       "    <th>0.005316</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>175</th>\n",
       "    <th>1.252276</th>\n",
       "    <th>0.537536</th>\n",
       "    <th>0.876500</th>\n",
       "    <th>0.667426</th>\n",
       "    <th>0.005848</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>176</th>\n",
       "    <th>1.143211</th>\n",
       "    <th>0.547173</th>\n",
       "    <th>0.878100</th>\n",
       "    <th>0.621906</th>\n",
       "    <th>0.005213</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>177</th>\n",
       "    <th>1.327963</th>\n",
       "    <th>0.522973</th>\n",
       "    <th>0.879200</th>\n",
       "    <th>0.684926</th>\n",
       "    <th>0.006430</th>\n",
       "    <th>04:33</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>178</th>\n",
       "    <th>1.123994</th>\n",
       "    <th>0.543591</th>\n",
       "    <th>0.879700</th>\n",
       "    <th>0.602794</th>\n",
       "    <th>0.005212</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>179</th>\n",
       "    <th>1.115683</th>\n",
       "    <th>0.554178</th>\n",
       "    <th>0.878800</th>\n",
       "    <th>0.604254</th>\n",
       "    <th>0.005114</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>180</th>\n",
       "    <th>1.183803</th>\n",
       "    <th>0.537698</th>\n",
       "    <th>0.877500</th>\n",
       "    <th>0.649395</th>\n",
       "    <th>0.005344</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>181</th>\n",
       "    <th>1.275013</th>\n",
       "    <th>0.533272</th>\n",
       "    <th>0.879100</th>\n",
       "    <th>0.660827</th>\n",
       "    <th>0.006142</th>\n",
       "    <th>04:25</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>182</th>\n",
       "    <th>1.172807</th>\n",
       "    <th>0.547360</th>\n",
       "    <th>0.880800</th>\n",
       "    <th>0.608189</th>\n",
       "    <th>0.005646</th>\n",
       "    <th>04:30</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>183</th>\n",
       "    <th>1.219460</th>\n",
       "    <th>0.535526</th>\n",
       "    <th>0.880500</th>\n",
       "    <th>0.639528</th>\n",
       "    <th>0.005799</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>184</th>\n",
       "    <th>1.230631</th>\n",
       "    <th>0.527883</th>\n",
       "    <th>0.879800</th>\n",
       "    <th>0.664875</th>\n",
       "    <th>0.005658</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>185</th>\n",
       "    <th>1.096366</th>\n",
       "    <th>0.541819</th>\n",
       "    <th>0.877900</th>\n",
       "    <th>0.602960</th>\n",
       "    <th>0.004934</th>\n",
       "    <th>04:31</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>186</th>\n",
       "    <th>1.173574</th>\n",
       "    <th>0.532132</th>\n",
       "    <th>0.880600</th>\n",
       "    <th>0.632472</th>\n",
       "    <th>0.005411</th>\n",
       "    <th>04:32</th>\n",
       "  </tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='355' class='' max='386', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      91.97% [355/386 04:03<00:21 1.1290]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(200,2e-3,wd=0.02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Training stopped at 186 epochs due to vm preemption."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "MixMatch clearly boasts impressive performance, but the downside is the additional time cost in training. Compared to the fully supervised case, training MixMatch takes approximately 2.5x longer. Some of this may be due to inefficiencies in the implementation, but generating multiple augmentations and then obtaining model predictions for labels has a significant cost, especially in the one GPU case. I trained the offical Tensorflow implemention for comparison, and verified that MixMatch takes a long time to fully converge; over twelve hours of training resulted in an error rate several percent higher than the one reported in the paper. It would take nearly two days of training on the P100 setup to match their results fully. However, a few hours of training will achieve the vast majority of accuracy improvement, with the final few percent taking most of the total training time.   \n",
    "\n",
    "While augmentation and sharpening are hugely beneficial, the paper's ablation study shows that the single most important component, error wise, is MixUp. This is also the most mysterious component in terms of why it works so well -- why should enforcing linearity in predictions between images help the model? Certainly it reduces memorization of training data, but so does data augmentation and to not nearly the same effect in this case. Even the original MixUp paper only provides informal arguments as to its efficacy; from that paper:\n",
    "\n",
    "> \"We argue that this linear behaviour reduces the amount of undesirable oscillations when predicting outside the\n",
    "training examples. Also, linearity is a good inductive bias from the perspective of Occam’s razor, since it is one of the simplest possible behaviors\" [4]\n",
    "\n",
    "Other researches have expanded upon the idea; for example by mixing up intermediate states instead of the input [7], or using a neural network instead of the beta function to generate the mixup coefficient [6]. However, I am unable to find a solid theoretical justification; this is yet another technique that falls into the 'it just works' category. Certainly it would be difficult to draw a biological analogy -- humans hardly learn a concept by blending it with an unrelated concept. \n",
    "\n",
    "That said, MixMatch is hugely promising and it will be interesting to see it applied to other domains beyond vision. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "[1]: Berthelot, David, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. “MixMatch: A Holistic Approach to Semi-Supervised Learning.” ArXiv:1905.02249 [Cs, Stat], May 6, 2019. http://arxiv.org/abs/1905.02249.\n",
    "\n",
    "[2]: Tarvainen, Antti, and Harri Valpola. “Mean Teachers Are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results.” ArXiv:1703.01780 [Cs, Stat], March 6, 2017. http://arxiv.org/abs/1703.01780.\n",
    "\n",
    "[3]: Miyato, Takeru, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning.” ArXiv:1704.03976 [Cs, Stat], April 12, 2017. http://arxiv.org/abs/1704.03976.\n",
    "\n",
    "[4]: Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. “Mixup: Beyond Empirical Risk Minimization.” ArXiv:1710.09412 [Cs, Stat], October 25, 2017. http://arxiv.org/abs/1710.09412.\n",
    "\n",
    "[5]: Polyak, Boris, and Anatoli Juditsky. “Acceleration of Stochastic Approximation by Averaging.” SIAM Journal on Control and Optimization 30 (July 1, 1992): 838–55. https://doi.org/10.1137/0330046.\n",
    "\n",
    "[6]: Guo, Hongyu, Yongyi Mao, and Richong Zhang. “MixUp as Locally Linear Out-Of-Manifold Regularization,” n.d., 9.\n",
    "\n",
    "[7]: Verma, Vikas, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. “Manifold Mixup: Better Representations by Interpolating Hidden States,” June 13, 2018. https://arxiv.org/abs/1806.05236v7."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
